Features - it's not a feature, it's a bug ;)
Jay
_Must Have_
- List processes
- Kill processes (cross-platform)
- Process information
* PID
* executable name / path
_Nice To Have_
- Process information
* Time info (user/kernel)
* Handle counts (windows - GetProcessHandleCount() )
* I/O counts (windows - GetProcessIoCounters() )
* Process environment variables (windows -
GetEnvironmentStrings() )
So, thoughts? Anything I missed? I tried to take a look at the Win32
API functions to get an idea for what kind of info is available, and I
compared with /proc on Linux and it looks like most of the info is
also obtainable there too.
http://msdn2.microsoft.com/en-us/library/ms802955.aspx
-Jay
Giampaolo Rodola'
- Process memory usage
- TCP/UDP ports opened by the process
- Process suspend / resume activity (what in UNIX is equal to "CTRL +
Z" -> "fg". On windows there's also a similar thing accessible by
using "procexp" utility)
- Kill process given its name
I created a wiki page in which I listed the tasks discussed until now:
http://code.google.com/p/psutil/wiki/Tasks
To modify it you have to use SVN since the web editor doesn't still
work.
--- Giampaolo
David Daeschler
scenarios:
- standard input, output, and error redirection to streams (could
probably use the implementation already in python [jay, what was the
name of that library that had i/o redirection?])
Nice to have:
- find processes by name
- Dave
David Daeschler
http://msdn2.microsoft.com/en-us/library/system.diagnostics.process_members.aspx
http://java.sun.com/j2se/1.5.0/docs/api/java/lang/ProcessBuilder.html
http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Process.html
http://docs.wxwidgets.org/stable/wx_wxprocess.html
- Dave
Jay Loden
David Daeschler wrote:
> Must haves if we want the "Process" object to be usable for more
> scenarios:
> - standard input, output, and error redirection to streams (could
> probably use the implementation already in python [jay, what was the
> name of that library that had i/o redirection?])
Are you talking about Java ProcessBuilder-like functionality? If so that's already done in Python via the subprocess module (which is cross platform) so we shouldn't need to worry about that. Or are you talking about redirecting stdout/stderr/stdin from an already running process?
> Nice to have:
> - find processes by name
I was planning on the lib also providing convenience methods, e.g.
- find process by name/PID
- kill process by name/PID
any other ones we can think of that it makes sense to have quick convenience methods for? The cool thing is these can probably be done in pure Python once we have a basic library implemented in C.
-Jay
Jay Loden
> Some ideas for the "Nice to have" list:
>
> - Process memory usage
> - TCP/UDP ports opened by the process
These are good suggestions also, and we could add to that "open files" in addition to open handles which I already mentioned. Also reminded me, thread counts could be useful information as well.
> - Process suspend / resume activity (what in UNIX is equal to "CTRL +
> Z" -> "fg". On windows there's also a similar thing accessible by
> using "procexp" utility)
I'm not sure about the UNIX side, because suspending with CTRL + Z is done as part of job control in the shell. There may not be a way to do this for processes that are already up and running and not started by the user in a shell. However, I've looked into this before on Windows. This sums it up pretty nicely:
http://www.codeproject.com/KB/threads/pausep.aspx
Also gave me another idea. We might just implement it as part of the "kill" function, but it probably makes sense to have a "quit()" method also that sends a process a shutdown message, e.g. WM_CLOSE on Windows or the UNIX "HUP".
David Daeschler
On Wed, 2008-04-23 at 10:51 -0400, Jay Loden wrote:
> Are you talking about Java ProcessBuilder-like functionality?
The way I was thinking is that the library would take any process
specific functionality and wrap it up in one place instead of having to
import 4 unrelated packages to do process work. Maybe the scope of the
idea is not as large as I thought, but I envisioned a process object
that could perform any common process related tasks..
An example:
process.py:
class Process: ...
def CreateProcess(...): returns new process
def OpenProcess(...): returns new process
proc = OpenProcess(PID)
proc.kill();
proc = OpenProcess(PID)
proc.stdIn.write("hi")
in the same way...
proc = CreateProcess(PARAMS)
proc.stdIn.write("hi")
If this is beyond the scope of the project, just let me know. I just
think it would be nice to unify common process functionality regardless
of who started the process (us or them)
- Dave
David Daeschler
On Wed, 2008-04-23 at 10:59 -0400, Jay Loden wrote:
>
> Also gave me another idea. We might just implement it as part of the
> "kill" function, but it probably makes sense to have a "quit()" method
> also that sends a process a shutdown message, e.g. WM_CLOSE on Windows
> or the UNIX "HUP".
Sounds like we should implement signal("") kind of like how the kill C
function takes a param for what signal to send (and it's not necessarily
SIGKILL)
man -S2 kill
- Dave
Jay Loden
Jay Loden
That's kind of what I had in mind under the hood, but with a convenience method of quit() or shutdown() wrapping it also. I don't know about you guys but I definitely never remember what the heck the different signals mean. Heck, I had to google it before I could remember HUP ;)
Giampaolo Rodola'
described in the wiki straight into the subprocess module instead of
creating a separated module?
In that case I think it could be a good idea but leaving subprocess
module alone, at least for now, would help us work more easily.
We still have to see which tasks are cross platform and which ones are
not.
I wouldn't like to have subprocess returning an object which has a
"list_threads" method on Windows but not on UNIX.
I'd first try to realize the level of compatibility of every task.
Once we'll have the list of tasks which are cross platform we'll see
if it will be worth to add them in the subprocess module or not.
Considering that almost every function described in the Tasks wiki
accepts a PID as argument, moving our stuff from "psutil" to
"subprocess" should be a piece of cake.
For example, at a first stage we'll have:
>>> import subprocess, psutil
>>> x = subprocess.Popen('gedit')
>>> psutil.kill(x.pid, 9)
>>>
...then:
>>> import subprocess
>>> x = subprocess.Popen('gedit')
>>> x.kill(9)
>>>
My2Cents
David Daeschler
On Wed, 2008-04-23 at 09:40 -0700, Giampaolo Rodola' wrote:
> Are you guys talking about adding the functionalities currently
> described in the wiki straight into the subprocess module instead of
> creating a separated module?
No. I think we still want to create a separate module, but I would like
to roll much of the functionality that exists in subprocess into our
psutil module. That way if a user wants to deal with processes at all,
they only have to use our module and not import 1, 2, or 3 others.
I want this library to be like the "Process" library found in other
languages/frameworks. Almost anything you would want to do with an OS
process should be doable through us without importing anything else.
The other illustration was simply showing how I wanted to work the
object interface for a process class in an example. Whether we started
the process, or we're dealing with a process already running (associated
through a PID) I think the interface should be the same.
- Dave
David Daeschler
On Wed, 2008-04-23 at 13:05 -0400, David Daeschler wrote:
> No. I think we still want to create a separate module, but I would
> like
> to roll much of the functionality that exists in subprocess into our
> psutil module.
An example pseudo UML is attached.
Note that to create a process with redirected standard input/output we
call into subprocess.Popen. This lets us use our process class for both
existing processes, and to create new processes with, for example,
redirected input streams. The bottom line is that the user never has to
learn subprocess, just psutil.
Giampaolo Rodola'
> object interface for a process class in an example. Whether we started
> the process, or we're dealing with a process already running (associated
> through a PID) I think the interface should be the same.
I think "OpenProcess(PID)" and "CreateProcess(PARAMS)" would be the
best design choice to guarantee the unique interface.
Should psutil.Popen() inherit from subprocess.Popen() or you want to
re-write the whole class from scratch?
Giampaolo Rodola'
> object interface for a process class in an example. Whether we started
> the process, or we're dealing with a process already running (associated
> through a PID) I think the interface should be the same.
I think "OpenProcess(PID)" and "CreateProcess(PARAMS)" would be the
best design choice to guarantee the unique interface.
Should psutil.Popen() inherit from subprocess.Popen() or you want to
re-write the whole class from scratch?
On 23 Apr, 19:05, David Daeschler <daver...@rsaisp.com> wrote:
David Daeschler
> Oh ok, I got it now. Sorry for the misunderstanding.
That's okay. Sometimes it's hard to properly express yourself in text
alone. It's not your fault.
> Should psutil.Popen() inherit from subprocess.Popen() or you want to
> re-write the whole class from scratch?
Unless you can think of a reason to inherit, I would write the class
from scrtch and then use subprocess.Popen as a dependency. I tend to
agree with the mantra "favor composition over inheritance"
(see http://www.artima.com/lejava/articles/designprinciples4.html)
We can still call into subprocess to do any work that they have already
done for us.
A HUGE oversimplification follows:
psutil.py:
def Popen(...):
return subprocess.Popen(...)
- Dave
Giampaolo Rodola'
If there's nothing else to decide I propose to start committing
something, at least to have an initial structure to start us off.
As for the module design I was thinking that we could use something
like this:
psutil
|
----- test
| |....... test_psutil.py
|
__init__.py
_win32.py
_posix.py
psutil.py
psutil.py would contain the Popen class and the proper code to have
Popen inherit from _win32 or _posix modules depending on the system it
is currently running on.
If you agree, since now I have a bunch of free hours off of work, I
would start writing some stuff for the _win32 module (I'd start trying
to implement at least list_process() and kill_process() functions
which should be easy to afford via the pywin32 extension).
Do any of you guys want to care about the work needed to be done in
psutil.Popen for the subprocess module integration?
--- Giampaolo
On 24 Apr, 13:43, David Daeschler <daver...@rsaisp.com> wrote:
> Hi Giampaolo,
>
> > Oh ok, I got it now. Sorry for the misunderstanding.
>
> That's okay. Sometimes it's hard to properly express yourself in text
> alone. It's not your fault.
>
> > Should psutil.Popen() inherit from subprocess.Popen() or you want to
> > re-write the whole class from scratch?
>
> Unless you can think of a reason to inherit, I would write the class
> from scrtch and then use subprocess.Popen as a dependency. I tend to
> agree with the mantra "favor composition over inheritance"
Giampaolo Rodola'
If there's nothing else to decide I propose to start committing
something, at least to have an initial structure to start us off.
As for the module design I was thinking that we could use something
like this:
psutil
|
----- test
| |....... test_psutil.py
|
__init__.py
_win32.py
_posix.py
psutil.py
psutil.py would contain the Popen class and the proper code to have
Popen inherit from _win32 or _posix modules depending on the system it
is currently running on.
If you agree, since now I have a bunch of free hours off of work, I
would start writing some stuff for the _win32 module (I'd start trying
to implement at least list_process() and kill_process() functions
which should be easy to afford via the pywin32 extension).
Do any of you guys want to care about the work needed to be done in
psutil.Popen for the subprocess module integration?
--- Giampaolo
On 24 Apr, 13:43, David Daeschler <daver...@rsaisp.com> wrote:
>
> > Oh ok, I got it now. Sorry for the misunderstanding.
>
> That's okay. Sometimes it's hard to properly express yourself in text
> alone. It's not your fault.
>
> > Should psutil.Popen() inherit from subprocess.Popen() or you want to
> > re-write the whole class from scratch?
>
> Unless you can think of a reason to inherit, I would write the class
> from scrtch and then use subprocess.Popen as a dependency. I tend to
> agree with the mantra "favor composition over inheritance"
Giampaolo Rodola'
If there's nothing else to decide I propose to start committing
something, at least to have an initial structure to start us off.
As for the module design I was thinking that we could use something
like this:
psutil
|
----- test
| |....... test_psutil.py
|
__init__.py
_win32.py
_posix.py
psutil.py
psutil.py would contain the Popen class and the proper code to have
Popen inherit from _win32 or _posix modules depending on the system it
is currently running on.
If you agree, since now I have a bunch of free hours off of work, I
would start writing some stuff for the _win32 module (I'd start trying
to implement at least list_process() and kill_process() functions
which should be easy to afford via the pywin32 extension).
Do any of you guys want to care about the work needed to be done in
psutil.Popen for the subprocess module integration?
--- Giampaolo
On 24 Apr, 13:43, David Daeschler <daver...@rsaisp.com> wrote:
>
> > Oh ok, I got it now. Sorry for the misunderstanding.
>
> That's okay. Sometimes it's hard to properly express yourself in text
> alone. It's not your fault.
>
> > Should psutil.Popen() inherit from subprocess.Popen() or you want to
> > re-write the whole class from scratch?
>
> Unless you can think of a reason to inherit, I would write the class
> from scrtch and then use subprocess.Popen as a dependency. I tend to
> agree with the mantra "favor composition over inheritance"
Giampaolo Rodola'
If there's nothing else to decide I propose to start committing
something, at least to have an initial structure to start us off.
As for the module design I was thinking that we could use something
like this:
psutil
|
----- test
| |....... test_psutil.py
|
__init__.py
_win32.py
_posix.py
psutil.py
psutil.py would contain the Popen class and the proper code to have
Popen inherit from _win32 or _posix modules depending on the system it
is currently running on.
If you agree, since now I have a bunch of free hours off of work, I
would start writing some stuff for the _win32 module (I'd start trying
to implement at least list_process() and kill_process() functions
which should be easy to afford via the pywin32 extension).
Do any of you guys want to care about the work needed to be done in
psutil.Popen for the subprocess module integration?
--- Giampaolo
On 24 Apr, 13:43, David Daeschler <daver...@rsaisp.com> wrote:
>
> > Oh ok, I got it now. Sorry for the misunderstanding.
>
> That's okay. Sometimes it's hard to properly express yourself in text
> alone. It's not your fault.
>
> > Should psutil.Popen() inherit from subprocess.Popen() or you want to
> > re-write the whole class from scratch?
>
> Unless you can think of a reason to inherit, I would write the class
> from scrtch and then use subprocess.Popen as a dependency. I tend to
> agree with the mantra "favor composition over inheritance"
Giampaolo Rodola'
If there's nothing else to decide I propose to start committing
something, at least to have an initial structure to start us off.
As for the module design I was thinking that we could use something
like this:
psutil
|
----- test
| |....... test_psutil.py
|
__init__.py
_win32.py
_posix.py
psutil.py
psutil.py would contain the Popen class and the proper code to have
Popen inherit from _win32 or _posix modules depending on the system it
is currently running on.
If you agree, since now I have a bunch of free hours off of work, I
would start writing some stuff for the _win32 module (I'd start trying
to implement at least list_process() and kill_process() functions
which should be easy to afford via the pywin32 extension).
Do any of you guys want to care about the work needed to be done in
psutil.Popen for the subprocess module integration?
--- Giampaolo
On 24 Apr, 13:43, David Daeschler <daver...@rsaisp.com> wrote:
>
> > Oh ok, I got it now. Sorry for the misunderstanding.
>
> That's okay. Sometimes it's hard to properly express yourself in text
> alone. It's not your fault.
>
> > Should psutil.Popen() inherit from subprocess.Popen() or you want to
> > re-write the whole class from scratch?
>
> Unless you can think of a reason to inherit, I would write the class
> from scrtch and then use subprocess.Popen as a dependency. I tend to
> agree with the mantra "favor composition over inheritance"
Giampaolo Rodola'
After having put hands on the initial code I wrote I realized that not
all tasks described in the wiki would fit well into an unique "Popen"
class.
For example, functions like:
list_processes()
kill_process_by_name(name)
get_process_pid(name)
...should be placed outside of Popen, hence I guess that the initial
design I proposed (see: import psutil; psutil.function(pid)) should be
kept for some tasks, IMO.
Since the Popen class still needs to be written in this first commit I
opted to follow an approach based on calling functions, without using
the "Popen.OpenProcess(PID)" approach proposed by David.
Example:
>>> import psutil
>>> psutil.list_processes()
[0, 4, 1108, 1180, 1208, 1252, 1264, 1432, 1444, 1528, 1968, 2000, 220, 296, 780
, 892, 1184, 1516, 1896, 436, 852, 828, 2056, 2276, 2300, 2320, 3796, 2964, 820,
3672, 3004, 4048, 1140, 1844, 2720, 3664, 2628, 3288, 1092, 3564]
>>> psutil.get_process_path(2720)
u'C:\\Programmi\\xchat\\xchat.exe'
>>> psutil.kill_process_by_name('xchat')
>>>
This API can be modified in a second time as soon as the Popen class
will be available, in which case we may decide whether offering an
unique Popen-class-based interface for doing a task:
>>> psutil.Popen().OpenProcess(PID).kill_process()
...or if also providing the corresponding function doing the same
thing separately:
>>> psutil.kill_process(PID)
I also wrote a very minimalistic test suite which tests all the
functions currently available in the _win32.py module.
I committed the whole stuff in revision #5
(http://code.google.com/p/psutil/source/detail?r=5) which includes:
- Implementation of 'list_processes', 'kill_process',
'kill_process_by_name', 'get_process_path' and
'get_process_name' functions for Windows system.
- a setup.py script.
- a minimalistic test suite.
Any thoughts is appreciated.
2008/4/26 Giampaolo Rodola' <billi...@gmail.com>:
> Sorry dudes,
> don't know why but the damn Google Groups seems to keep sending my
> last message over and over again even if I sent it just once by using
> the Google Groups web interface.
> Now I'm sending this message by using SMTP.
> I'm going to investigate to see if I can understand what the hell is going on.
>
>
> 2008/4/26 Giampaolo Rodola' <billi...@gmail.com>:
Jay Loden
> Hey there guys.
> If there's nothing else to decide I propose to start committing
> something, at least to have an initial structure to start us off.
> As for the module design I was thinking that we could use something
> like this:
>
> psutil
> |
> ----- test
> | |....... test_psutil.py
> |
> __init__.py
> _win32.py
> _posix.py
> psutil.py
We will need the structure of the module to take into account the C code modules, which would go in a _psutil.so / _psutil.lib file. The actual module should only be "psutil.py" and the C module, which would the _psutil.so file. In SVN I would suggest just having a test directory for the test scripts, and the psutil.py in the root. Until we decide how to structure the C code we can't really design a source repository for it, so I'd just add any C files later.
> psutil.py would contain the Popen class and the proper code to have
> Popen inherit from _win32 or _posix modules depending on the system it
> is currently running on.
This is unnecessary, all platform-specific code will be in the C module, so our Python code would just "import _psutil" and we make sure that we've compiled the correct C code for the platform we are on. We shouldn't need any Python files beyond psutil.py, unless we decide to separate some of the classes into separate files for easier maintenance.
> If you agree, since now I have a bunch of free hours off of work, I
> would start writing some stuff for the _win32 module (I'd start trying
> to implement at least list_process() and kill_process() functions
> which should be easy to afford via the pywin32 extension).
One of the main things I *don't* want to do is use the pywin32 extensions because they're not part of the standard library and it will force us to have a dependency on a third-party module. By using C code to implement the platform-specific details it will ensure that we have a single consistent module that is imported and needs no dependencies on other libraries. The _psutil lib is just compiled with the platform specific code, and when psutil.py imports _psutil, it will automatically be using the right code.
> Do any of you guys want to care about the work needed to be done in
> psutil.Popen for the subprocess module integration?
I think we need to design the API first, before we can do the coding part...we need to decide what classes will be used to represent processes and other objects, then design our subprocess module integration around that. Let's start with getting an API specification set up. Have we decided on a final list of features? Let's make sure we have our official list of features and then we'll create the API specification so we can get to coding.
I am also looking into some tools, it looks like there's a number of projects out there that help you wrap your C code with the Python extensions automatically. So if we just write our C functions the bulk of the tedious work with the Python C extensions could be done for us by something like SIP, which is good news for Dave and I :-)
-Jay
Giampaolo Rodola'
I agree we shouldn't use pywin32 or other dependencies.
> I think we need to design the API first, before we can do the
> coding part...we need to decide what classes will be used to
> represent processes and other objects, then design our
> subprocess module integration around that. Let's start with
> getting an API specification set up. Have we decided on a final
> list of features? Let's make sure we have our official list of
> features and then we'll create the API specification so we can
> get to coding.
As far as I understood the API should be the same as subprocess.Popen
with in addition the open/create process methods and the tasks
described in the wiki, isn't it?
2008/4/27 Jay Loden <jlo...@gmail.com>:
Jay Loden
>> I think we need to design the API first, before we can do the
>> coding part...we need to decide what classes will be used to
>> represent processes and other objects, then design our
>> subprocess module integration around that. Let's start with
>> getting an API specification set up. Have we decided on a final
>> list of features? Let's make sure we have our official list of
>> features and then we'll create the API specification so we can
>> get to coding.
>
> As far as I understood the API should be the same as subprocess.Popen
> with in addition the open/create process methods and the tasks
> described in the wiki, isn't it?
No, we didn't want to copy the subprocess module style...we wanted to create an API that makes the most logical sense for process management, whatever we decide that should be. Then once we have our API designed and implemented, we want to wrap subprocess module functionality so that you can use our API/interface for all process management functions, whether it's listing running processes, starting a new process, killing a process, etc.
For example, let's say our API has a Process object that represents all information about a given process in named attributes. Here is a fake code sample just as an example, not necessarily the actual API we end up with:
#!/usr/bin/python
import time
import psutil
# list all processes' executable name and memory usage
for myProcess in psutil.processiter():
exe = myProcess.ps_name
mem = (myProcess.ps_totalMemory / 1024)
print "Process %s is using %s MB of memory" % (os.path.basename(exe), mem)
# create a new process then list some information about it
newProcess = psutil.Popen(["myProgram.exe", "--arg1=foo", "--arg2=bar"])
print "Started new process %s, currently using %sMB of memory" % (newProcess.ps_name, (newProcess.ps_totalMemory / 1024))
# kill the new started application with a SIGTERM signal after waiting 5 seconds
time.sleep(5)
newProcess.kill(psutil.SIGTERM)
So the Popen function would be the same as in subprocess (same arguments and would call subprocess.Popen underneath, but then it would return a psutil.Process object with all the same attributes and methods. Make sense?
-Jay
David Daeschler
I agree the design needs to be done before we start any real code. We
should probably start with a simple requirements doc to set in stone
exactly what we want the library to do.
An example of a (very verbose) requirements document can be found here:
http://www-wnt.gsi.de/GO4/Docs/go4_urd_201.html
Ours will be much shorter.
- Dave