Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Python child process in while True loop blocks parent
96 views
Skip to first unread message
Jen Kris
Nov 29, 2021, 3:34:38 PM11/29/21
to
I have a C program that forks to create a child process and uses execv to call a Python program. The Python program communicates with the parent process (in C) through a FIFO pipe monitored with epoll().
The Python child process is in a while True loop, which is intended to keep it running while the parent process proceeds, and perform functions for the C program only at intervals when the parent sends data to the child -- similar to a daemon process.
The C process writes to its end of the pipe and the child process reads it, but then the child process continues to loop, thereby blocking the parent.
This is the Python code:
#!/usr/bin/python3
import os
import select
#Open the named pipes
pr = os.open('/tmp/Pipe_01', os.O_RDWR)
pw = os.open('/tmp/Pipe_02', os.O_RDWR)
ep = select.epoll(-1)
ep.register(pr, select.EPOLLIN)
while True:
events = ep.poll(timeout=2.5, maxevents=-1)
#events = ep.poll(timeout=None, maxevents=-1)
print("child is looping")
for fileno, event in events:
print("Python fileno")
print(fileno)
print("Python event")
print(event)
v = os.read(pr,64)
print("Pipe value")
print(v)
The child process correctly receives the signal from ep.poll and correctly reads the data in the pipe, but then it continues looping. For example, when I put in a timeout:
child is looping
Python fileno
4
Python event
1
Pipe value
b'10\x00'
child is looping
child is looping
That suggests that a while True loop is not the right thing to do in this case. My question is, what type of process loop is best for this situation? The multiprocessing, asyncio and subprocess libraries are very extensive, and it would help if someone could suggest the best alternative for what I am doing here.
Thanks very much for any ideas.
The Python child process is in a while True loop, which is intended to keep it running while the parent process proceeds, and perform functions for the C program only at intervals when the parent sends data to the child -- similar to a daemon process.
The C process writes to its end of the pipe and the child process reads it, but then the child process continues to loop, thereby blocking the parent.
This is the Python code:
#!/usr/bin/python3
import os
import select
#Open the named pipes
pr = os.open('/tmp/Pipe_01', os.O_RDWR)
pw = os.open('/tmp/Pipe_02', os.O_RDWR)
ep = select.epoll(-1)
ep.register(pr, select.EPOLLIN)
while True:
events = ep.poll(timeout=2.5, maxevents=-1)
#events = ep.poll(timeout=None, maxevents=-1)
print("child is looping")
for fileno, event in events:
print("Python fileno")
print(fileno)
print("Python event")
print(event)
v = os.read(pr,64)
print("Pipe value")
print(v)
The child process correctly receives the signal from ep.poll and correctly reads the data in the pipe, but then it continues looping. For example, when I put in a timeout:
child is looping
Python fileno
4
Python event
1
Pipe value
b'10\x00'
child is looping
child is looping
That suggests that a while True loop is not the right thing to do in this case. My question is, what type of process loop is best for this situation? The multiprocessing, asyncio and subprocess libraries are very extensive, and it would help if someone could suggest the best alternative for what I am doing here.
Thanks very much for any ideas.
Cameron Simpson
Nov 29, 2021, 4:11:27 PM11/29/21
to
On 29Nov2021 21:34, Jen Kris <jen...@tutanota.com> wrote:
>I have a C program that forks to create a child process and uses execv to call a Python program. The Python program communicates with the parent process (in C) through a FIFO pipe monitored with epoll().
>
>The Python child process is in a while True loop, which is intended to keep it running while the parent process proceeds, and perform functions for the C program only at intervals when the parent sends data to the child -- similar to a daemon process.
>
>The C process writes to its end of the pipe and the child process reads it, but then the child process continues to loop, thereby blocking the parent.
It seems to me that the child Python process never writes anything back
>I have a C program that forks to create a child process and uses execv to call a Python program. The Python program communicates with the parent process (in C) through a FIFO pipe monitored with epoll().
>
>The Python child process is in a while True loop, which is intended to keep it running while the parent process proceeds, and perform functions for the C program only at intervals when the parent sends data to the child -- similar to a daemon process.
>
>The C process writes to its end of the pipe and the child process reads it, but then the child process continues to loop, thereby blocking the parent.
to the parent. If the parent is waiting for some response, of course it
will be blocked.
Personally I wouldn't be using epoll at all. I'd just read data from pr,
act on it, and write back to pw. That way the child blocks waiting for
the parent istead of polling. You only want epoll if you're either
polling something _while_ you do other work, or monitoring more than one
thing. A plain read-from-one-pipe, work, write-back-to-the-other does
not need it.
Cheers,
Cameron Simpson <c...@cskk.id.au>
Barry
Nov 29, 2021, 5:12:44 PM11/29/21
to
> On 29 Nov 2021, at 20:36, Jen Kris via Python-list <pytho...@python.org> wrote:
>
> I have a C program that forks to create a child process and uses execv to call a Python program. The Python program communicates with the parent process (in C) through a FIFO pipe monitored with epoll().
>
> The Python child process is in a while True loop, which is intended to keep it running while the parent process proceeds, and perform functions for the C program only at intervals when the parent sends data to the child -- similar to a daemon process.
>
> The C process writes to its end of the pipe and the child process reads it, but then the child process continues to loop, thereby blocking the parent.
>
> This is the Python code:
>
> #!/usr/bin/python3
> import os
> import select
>
> #Open the named pipes
> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
Why open rdwr if you are only going to read the pipe?
>
> #!/usr/bin/python3
> import os
> import select
>
> #Open the named pipes
> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
> pw = os.open('/tmp/Pipe_02', os.O_RDWR)
Only need to open for write.
>
> ep = select.epoll(-1)
> ep.register(pr, select.EPOLLIN)
Is the only thing that the child does this:
> ep = select.epoll(-1)
> ep.register(pr, select.EPOLLIN)
1. Read message from pr
2. Process message
3. Write result to pw.
4. Loop from 1
If so as Cameron said you do not need to worry about the poll.
Do you plan for the child to become more complex?
>
> while True:
>
> events = ep.poll(timeout=2.5, maxevents=-1)
> #events = ep.poll(timeout=None, maxevents=-1)
>
> print("child is looping")
>
> for fileno, event in events:
> print("Python fileno")
> print(fileno)
> print("Python event")
> print(event)
> v = os.read(pr,64)
> print("Pipe value")
> print(v)
>
> The child process correctly receives the signal from ep.poll and correctly reads the data in the pipe, but then it continues looping. For example, when I put in a timeout:
>
> child is looping
> Python fileno
> 4
> Python event
> 1
> Pipe value
> b'10\x00'
I assume the 0 is from the end of a C string.
UDS messages have a length.
In the C just write 2 byes in the case.
Barry
> child is looping
> child is looping
>
> That suggests that a while True loop is not the right thing to do in this case. My question is, what type of process loop is best for this situation? The multiprocessing, asyncio and subprocess libraries are very extensive, and it would help if someone could suggest the best alternative for what I am doing here.
>
> Thanks very much for any ideas.
>
>
> https://mail.python.org/mailman/listinfo/python-list
Barry
Nov 29, 2021, 5:14:59 PM11/29/21
to
> On 29 Nov 2021, at 20:36, Jen Kris via Python-list <pytho...@python.org> wrote:
>
> I have a C program that forks to create a child process and uses execv to call a Python program. The Python program communicates with the parent process (in C) through a FIFO pipe monitored with epoll().
>
> The Python child process is in a while True loop, which is intended to keep it running while the parent process proceeds, and perform functions for the C program only at intervals when the parent sends data to the child -- similar to a daemon process.
>
> The C process writes to its end of the pipe and the child process reads it, but then the child process continues to loop, thereby blocking the parent.
>
> This is the Python code:
>
> #!/usr/bin/python3
> import os
> import select
>
> #Open the named pipes
> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
> pw = os.open('/tmp/Pipe_02', os.O_RDWR)
You will need to set the fd’s to non blocking on parent and child.
Otherwise the parent will block on its write until the child reads the message.
Barry
>
> ep = select.epoll(-1)
> ep.register(pr, select.EPOLLIN)
>
> ep = select.epoll(-1)
> ep.register(pr, select.EPOLLIN)
>
> while True:
>
> events = ep.poll(timeout=2.5, maxevents=-1)
> #events = ep.poll(timeout=None, maxevents=-1)
>
> print("child is looping")
>
> for fileno, event in events:
> print("Python fileno")
> print(fileno)
> print("Python event")
> print(event)
> v = os.read(pr,64)
> print("Pipe value")
> print(v)
>
> The child process correctly receives the signal from ep.poll and correctly reads the data in the pipe, but then it continues looping. For example, when I put in a timeout:
>
> child is looping
> Python fileno
> 4
> Python event
> 1
> Pipe value
> b'10\x00'
>
> events = ep.poll(timeout=2.5, maxevents=-1)
> #events = ep.poll(timeout=None, maxevents=-1)
>
> print("child is looping")
>
> for fileno, event in events:
> print("Python fileno")
> print(fileno)
> print("Python event")
> print(event)
> v = os.read(pr,64)
> print("Pipe value")
> print(v)
>
> The child process correctly receives the signal from ep.poll and correctly reads the data in the pipe, but then it continues looping. For example, when I put in a timeout:
>
> child is looping
> Python fileno
> 4
> Python event
> 1
> Pipe value
> b'10\x00'
Jen Kris
Nov 29, 2021, 5:32:12 PM11/29/21
to
Thanks to you and Cameron for your replies. The C side has an epoll_ctl set, but no event loop to handle it yet. I'm putting that in now with a pipe write in Python-- as Cameron pointed out that is the likely source of blocking on C. The pipes are opened as rdwr in Python because that's nonblocking by default. The child will become more complex, but not in a way that affects polling. And thanks for the tip about the c-string termination.
Nov 29, 2021, 14:12 by ba...@barrys-emacs.org:
>
>
>> On 29 Nov 2021, at 20:36, Jen Kris via Python-list <pytho...@python.org> wrote:
>>
>> I have a C program that forks to create a child process and uses execv to call a Python program. The Python program communicates with the parent process (in C) through a FIFO pipe monitored with epoll().
>>
>> The Python child process is in a while True loop, which is intended to keep it running while the parent process proceeds, and perform functions for the C program only at intervals when the parent sends data to the child -- similar to a daemon process.
>>
>> The C process writes to its end of the pipe and the child process reads it, but then the child process continues to loop, thereby blocking the parent.
>>
>> This is the Python code:
>>
>> #!/usr/bin/python3
>> import os
>> import select
>>
>> #Open the named pipes
>> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
>>
Nov 29, 2021, 14:12 by ba...@barrys-emacs.org:
>
>
>> On 29 Nov 2021, at 20:36, Jen Kris via Python-list <pytho...@python.org> wrote:
>>
>> I have a C program that forks to create a child process and uses execv to call a Python program. The Python program communicates with the parent process (in C) through a FIFO pipe monitored with epoll().
>>
>> The Python child process is in a while True loop, which is intended to keep it running while the parent process proceeds, and perform functions for the C program only at intervals when the parent sends data to the child -- similar to a daemon process.
>>
>> The C process writes to its end of the pipe and the child process reads it, but then the child process continues to loop, thereby blocking the parent.
>>
>> This is the Python code:
>>
>> #!/usr/bin/python3
>> import os
>> import select
>>
>> #Open the named pipes
>> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
>>
> Why open rdwr if you are only going to read the pipe?
>
>
>> pw = os.open('/tmp/Pipe_02', os.O_RDWR)
>>
>>
> Only need to open for write.
>
>>
>
>>
>> ep = select.epoll(-1)
>> ep.register(pr, select.EPOLLIN)
>>
>
>> ep.register(pr, select.EPOLLIN)
>>
>
> Is the only thing that the child does this:
> 1. Read message from pr
> 2. Process message
> 3. Write result to pw.
> 4. Loop from 1
>
> If so as Cameron said you do not need to worry about the poll.
> Do you plan for the child to become more complex?
>
>>
> 1. Read message from pr
> 2. Process message
> 3. Write result to pw.
> 4. Loop from 1
>
> If so as Cameron said you do not need to worry about the poll.
> Do you plan for the child to become more complex?
>
>>
>> while True:
>>
>> events = ep.poll(timeout=2.5, maxevents=-1)
>> #events = ep.poll(timeout=None, maxevents=-1)
>>
>> print("child is looping")
>>
>> for fileno, event in events:
>> print("Python fileno")
>> print(fileno)
>> print("Python event")
>> print(event)
>> v = os.read(pr,64)
>> print("Pipe value")
>> print(v)
>>
>> The child process correctly receives the signal from ep.poll and correctly reads the data in the pipe, but then it continues looping. For example, when I put in a timeout:
>>
>> child is looping
>> Python fileno
>> 4
>> Python event
>> 1
>> Pipe value
>> b'10\x00'
>>
>>
>> events = ep.poll(timeout=2.5, maxevents=-1)
>> #events = ep.poll(timeout=None, maxevents=-1)
>>
>> print("child is looping")
>>
>> for fileno, event in events:
>> print("Python fileno")
>> print(fileno)
>> print("Python event")
>> print(event)
>> v = os.read(pr,64)
>> print("Pipe value")
>> print(v)
>>
>> The child process correctly receives the signal from ep.poll and correctly reads the data in the pipe, but then it continues looping. For example, when I put in a timeout:
>>
>> child is looping
>> Python fileno
>> 4
>> Python event
>> 1
>> Pipe value
>> b'10\x00'
>>
> The C code does not need to write a 0 bytes at the end.
> I assume the 0 is from the end of a C string.
> UDS messages have a length.
> In the C just write 2 byes in the case.
>
> Barry
>
> I assume the 0 is from the end of a C string.
> UDS messages have a length.
> In the C just write 2 byes in the case.
>
> Barry
>
Barry Scott
Nov 30, 2021, 2:42:38 PM11/30/21
to
> On 29 Nov 2021, at 22:31, Jen Kris <jen...@tutanota.com> wrote:
>
> Thanks to you and Cameron for your replies. The C side has an epoll_ctl set, but no event loop to handle it yet. I'm putting that in now with a pipe write in Python-- as Cameron pointed out that is the likely source of blocking on C. The pipes are opened as rdwr in Python because that's nonblocking by default. The child will become more complex, but not in a way that affects polling. And thanks for the tip about the c-string termination.
>
You should not use O_RDWR when you only need O_RDONLY access or only O_WRONLY access.
You may find
man 2 open
useful to understand in detail what is behind os.open().
Barry
Jen Kris
Dec 1, 2021, 11:02:03 AM12/1/21
to
Thanks for your comment re blocking.
I removed pipes from the Python and C programs to see if it blocks without them, and it does. It looks now like the problem is not pipes. I use fork() and execv() in C to run Python in a child process, but the Python process blocks because fork() does not create a new thread, so the Python global interpreter lock (GIL) prevents the C program from running once Python starts. So the solution appears to be run Python in a separate thread, which I can do with pthread create. See "Thread State and the Global Interpreter Lock" https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock and the sections below that "Non-Python created threads" and "Cautions about fork()."
I'm working on that today and I hope all goes well :)
Nov 30, 2021, 11:42 by ba...@barrys-emacs.org:
I removed pipes from the Python and C programs to see if it blocks without them, and it does. It looks now like the problem is not pipes. I use fork() and execv() in C to run Python in a child process, but the Python process blocks because fork() does not create a new thread, so the Python global interpreter lock (GIL) prevents the C program from running once Python starts. So the solution appears to be run Python in a separate thread, which I can do with pthread create. See "Thread State and the Global Interpreter Lock" https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock and the sections below that "Non-Python created threads" and "Cautions about fork()."
I'm working on that today and I hope all goes well :)
Nov 30, 2021, 11:42 by ba...@barrys-emacs.org:
Barry Scott
Dec 4, 2021, 12:22:47 PM12/4/21
to
> On 1 Dec 2021, at 16:01, Jen Kris <jen...@tutanota.com> wrote:
>
> Thanks for your comment re blocking.
>
> I removed pipes from the Python and C programs to see if it blocks without them, and it does.
> It looks now like the problem is not pipes.
> I use fork() and execv() in C to run Python in a child process, but the Python process blocks
You will need to use the option to follow subprocesses so that you can see what goes on in the python process.
See man strace and the --follow-forks and --output-separately options.
That will allow you to find the blocking system call that your code is making.
> because fork() does not create a new thread, so the Python global interpreter lock (GIL) prevents the C program from running once Python starts.
> So the solution appears to be run Python in a separate thread, which I can do with pthread create.
I take it you mean that in the parent you think that using pthreads will affect python after the exec() call?
I does not. After exec() the process has one main thread create by the kernel and a new address space as defined by the /usr/bin/python.
The only state that in inherited from the parent are open file descriptors, the current working directory and security state like UID, GID.
> I'm working on that today and I hope all goes well :)
Maybe "Advanced Programming in the UNIX Environment" would be helpful?
https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk <https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk>
It's a great book and covers a wide range of Unix systems programming topics.
Have you created a small C program that just does the fork and exec of a python program to test out your assumptions?
If not I recommend that you do.
Barry
>
>
>
> Nov 30, 2021, 11:42 by ba...@barrys-emacs.org:
>
>
>> On 29 Nov 2021, at 22:31, Jen Kris <jen...@tutanota.com <mailto:jen...@tutanota.com>> wrote:
>>
>> Thanks to you and Cameron for your replies. The C side has an epoll_ctl set, but no event loop to handle it yet. I'm putting that in now with a pipe write in Python-- as Cameron pointed out that is the likely source of blocking on C. The pipes are opened as rdwr in Python because that's nonblocking by default. The child will become more complex, but not in a way that affects polling. And thanks for the tip about the c-string termination.
>>
>
> flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK.
> You should not use O_RDWR when you only need O_RDONLY access or only O_WRONLY access.
>
> You may find
>
> man 2 open
>
> useful to understand in detail what is behind os.open().
>
> Barry
>
>
>
>>
>>
>>
>
>
Jen Kris
Dec 5, 2021, 12:54:56 PM12/5/21
to
Thanks for your comments.
I put the Python program on its own pthread, and call a small C program to fork-execv to call the Python program as a child process. I revised the Python program to be a multiprocessing loop using the Python multiprocessing module. That bypasses the GIL and allows Python to run concurrently with C. So far so good.
Next I will use Linux pipes, not Python multiprocessing pipes, for IPC between Python and C. Multiprocessing pipes are (as far as I can tell) only for commo between two Python processes. I will have the parent thread send a signal through the pipe to the child process to exit when the parent thread is ready to exit, then call wait() to finalize the child process.
I will reply back when it's finished and post the code so you can see what I have done.
Thanks again.
Jen
Dec 4, 2021, 09:22 by ba...@barrys-emacs.org:
>
>
>> On 1 Dec 2021, at 16:01, Jen Kris <>> jen...@tutanota.com>> > wrote:
>>
>> Thanks for your comment re blocking.
>>
>> I removed pipes from the Python and C programs to see if it blocks without them, and it does.
>>
>> It looks now like the problem is not pipes.
>>
>
> Ok.
>
>
>> I use fork() and execv() in C to run Python in a child process, but the Python process blocks
>>
>
> Use strace on the parent process to see what is happening.
> You will need to use the option to follow subprocesses so that you can see what goes on in the python process.
>
> See man strace and the --follow-forks and --output-separately options.
> That will allow you to find the blocking system call that your code is making.
>
>
>> because fork() does not create a new thread, so the Python global interpreter lock (GIL) prevents the C program from running once Python starts.
>>
>
> Not sure why you think this.
>
>
>> So the solution appears to be run Python in a separate thread, which I can do with pthread create.
>>
>> See "Thread State and the Global Interpreter Lock" >> https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock>> and the sections below that "Non-Python created threads" and "Cautions about fork()."
I put the Python program on its own pthread, and call a small C program to fork-execv to call the Python program as a child process. I revised the Python program to be a multiprocessing loop using the Python multiprocessing module. That bypasses the GIL and allows Python to run concurrently with C. So far so good.
Next I will use Linux pipes, not Python multiprocessing pipes, for IPC between Python and C. Multiprocessing pipes are (as far as I can tell) only for commo between two Python processes. I will have the parent thread send a signal through the pipe to the child process to exit when the parent thread is ready to exit, then call wait() to finalize the child process.
I will reply back when it's finished and post the code so you can see what I have done.
Thanks again.
Jen
Dec 4, 2021, 09:22 by ba...@barrys-emacs.org:
>
>
>> On 1 Dec 2021, at 16:01, Jen Kris <>> jen...@tutanota.com>> > wrote:
>>
>> Thanks for your comment re blocking.
>>
>> I removed pipes from the Python and C programs to see if it blocks without them, and it does.
>>
>> It looks now like the problem is not pipes.
>>
>
> Ok.
>
>
>> I use fork() and execv() in C to run Python in a child process, but the Python process blocks
>>
>
> Use strace on the parent process to see what is happening.
> You will need to use the option to follow subprocesses so that you can see what goes on in the python process.
>
> See man strace and the --follow-forks and --output-separately options.
> That will allow you to find the blocking system call that your code is making.
>
>
>> because fork() does not create a new thread, so the Python global interpreter lock (GIL) prevents the C program from running once Python starts.
>>
>
> Not sure why you think this.
>
>
>> So the solution appears to be run Python in a separate thread, which I can do with pthread create.
>>
>>
>
> I take it you mean that in the parent you think that using pthreads will affect python after the exec() call?
> I does not. After exec() the process has one main thread create by the kernel and a new address space as defined by the /usr/bin/python.
> The only state that in inherited from the parent are open file descriptors, the current working directory and security state like UID, GID.
>
>
>> I'm working on that today and I hope all goes well :)
>>
>
> You seem to be missing background information on how processes work.
> Maybe "Advanced Programming in the UNIX Environment" > would be helpful?
>
> https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk>
>
>
> I take it you mean that in the parent you think that using pthreads will affect python after the exec() call?
> I does not. After exec() the process has one main thread create by the kernel and a new address space as defined by the /usr/bin/python.
> The only state that in inherited from the parent are open file descriptors, the current working directory and security state like UID, GID.
>
>
>> I'm working on that today and I hope all goes well :)
>>
>
> You seem to be missing background information on how processes work.
> Maybe "Advanced Programming in the UNIX Environment" > would be helpful?
>
> https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk>
>
> It's a great book and covers a wide range of Unix systems programming topics.
>
> Have you created a small C program that just does the fork and exec of a python program to test out your assumptions?
> If not I recommend that you do.
>
> Barry
>
>
>
>>
>>
>>
>> Nov 30, 2021, 11:42 by >> ba...@barrys-emacs.org>> :
>>
>>>
>>>
>>>
>>>> On 29 Nov 2021, at 22:31, Jen Kris <>>>> jen...@tutanota.com>>>> > wrote:
>>>>
>>>> Thanks to you and Cameron for your replies. The C side has an epoll_ctl set, but no event loop to handle it yet. I'm putting that in now with a pipe write in Python-- as Cameron pointed out that is the likely source of blocking on C. The pipes are opened as rdwr in Python because that's nonblocking by default. The child will become more complex, but not in a way that affects polling. And thanks for the tip about the c-string termination.
>>>>
>>>>
>>>
>>> flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK.
>>> You should not use O_RDWR when you only need O_RDONLY access or only O_WRONLY access.
>>>
>>> You may find
>>>
>>> man 2 open
>>>
>>> useful to understand in detail what is behind os.open().
>>>
>>> Barry
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>> Nov 29, 2021, 14:12 by >>>> ba...@barrys-emacs.org>>>> :
>
> Have you created a small C program that just does the fork and exec of a python program to test out your assumptions?
> If not I recommend that you do.
>
> Barry
>
>
>
>>
>>
>>
>> Nov 30, 2021, 11:42 by >> ba...@barrys-emacs.org>> :
>>
>>>
>>>
>>>
>>>> On 29 Nov 2021, at 22:31, Jen Kris <>>>> jen...@tutanota.com>>>> > wrote:
>>>>
>>>> Thanks to you and Cameron for your replies. The C side has an epoll_ctl set, but no event loop to handle it yet. I'm putting that in now with a pipe write in Python-- as Cameron pointed out that is the likely source of blocking on C. The pipes are opened as rdwr in Python because that's nonblocking by default. The child will become more complex, but not in a way that affects polling. And thanks for the tip about the c-string termination.
>>>>
>>>>
>>>
>>> flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK.
>>> You should not use O_RDWR when you only need O_RDONLY access or only O_WRONLY access.
>>>
>>> You may find
>>>
>>> man 2 open
>>>
>>> useful to understand in detail what is behind os.open().
>>>
>>> Barry
>>>
>>>
>>>
>>>
>>>>
>>>>
Barry
Dec 5, 2021, 6:20:19 PM12/5/21
to
> On 5 Dec 2021, at 17:54, Jen Kris <jen...@tutanota.com> wrote:
>
>
> Thanks for your comments.
>
> I put the Python program on its own pthread, and call a small C program to fork-execv to call the Python program as a child process.
What do you mean by putting python in it’s own pthread?
>
> I put the Python program on its own pthread, and call a small C program to fork-execv to call the Python program as a child process.
Are you embedding python in an other program?
Barry
Jen Kris
Dec 5, 2021, 6:51:34 PM12/5/21
to
By embedding, I think you may be referring to embedding Python in a C program with the Python C API. That's not what I'm doing here -- I'm not using the Python C API. The C program creates two threads (using pthreads), one for itself and one for the child process. On creation, the second pthread is pointed to a C program that calls fork-execv to run the Python program. That way Python runs on a separate thread. The multiprocessing library "effectively side-step[s] the Global Interpreter Lock by using subprocesses instead of threads." https://docs.python.org/3/library/multiprocessing.html. This way I can get the Python functionality I want on call from the C program through pipes and shared memory.
I don't want to use the C API because I will be making certain library calls from the C program, and the syntax is much easier with native Python code than with C API code.
I hope that clarifies what I'm doing.
Jen
Dec 5, 2021, 15:19 by ba...@barrys-emacs.org:
I don't want to use the C API because I will be making certain library calls from the C program, and the syntax is much easier with native Python code than with C API code.
I hope that clarifies what I'm doing.
Jen
Dec 5, 2021, 15:19 by ba...@barrys-emacs.org:
Peter J. Holzer
Dec 5, 2021, 9:17:37 PM12/5/21
to
On 2021-12-06 00:51:13 +0100, Jen Kris via Python-list wrote:
> The C program creates two threads (using pthreads), one for itself and
> one for the child process. On creation, the second pthread is pointed
> to a C program that calls fork-execv to run the Python program. That
> way Python runs on a separate thread.
I think you have the relationship between processes and threads
> The C program creates two threads (using pthreads), one for itself and
> one for the child process. On creation, the second pthread is pointed
> to a C program that calls fork-execv to run the Python program. That
> way Python runs on a separate thread.
backwards. A process consists of one or more threads. Fork creates a new
process and therefore also a new thread.
hp
--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | h...@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"
Barry
Dec 6, 2021, 3:22:47 AM12/6/21
to
> On 5 Dec 2021, at 23:51, Jen Kris <jen...@tutanota.com> wrote:
>
>
> By embedding, I think you may be referring to embedding Python in a C program with the Python C API. That's not what I'm doing here -- I'm not using the Python C API. The C program creates two threads (using pthreads), one for itself and one for the child process. On creation, the second pthread is pointed to a C program that calls fork-execv to run the Python program. That way Python runs on a separate thread. The multiprocessing library "effectively side-step[s] the Global Interpreter Lock by using subprocesses instead of threads." https://docs.python.org/3/library/multiprocessing.html. This way I can get the Python functionality I want on call from the C program through pipes and shared memory.
>
> I don't want to use the C API because I will be making certain library calls from the C program, and the syntax is much easier with native Python code than with C API code.
>
> I hope that clarifies what I'm doing.
You are confusing the threading needs of a python program and extension written in C
> I don't want to use the C API because I will be making certain library calls from the C program, and the syntax is much easier with native Python code than with C API code.
>
> I hope that clarifies what I'm doing.
and what you need to do to start a child process.
I see no benifit from calling fork and exec from a new thread. Indeed you may
encounter more issues.
What ever the bug is that you and trying to fix threading has nothing to do with it.
What if you change you code to run bash script instead of python?
Does you C program block them?
What did strace tell you was the reason that your C parent program blocked?
Barry
Jen Kris
Dec 6, 2021, 12:09:56 PM12/6/21
to
I can't find any support for your comment that "Fork creates a new
process and therefore also a new thread." From the Linux man pages https://www.man7.org/linux/man-pages/man2/fork.2.html, "The child process is created with a single thread—the one that called fork()."
I have a one-core one-thread instance at Digital Ocean available running Ubuntu 18.04. I can fork and create a new process on it, but it doesn't create a new thread because it doesn't have one available.
You may also want to see "Forking vs Threading" (https://www.geekride.com/fork-forking-vs-threading-thread-linux-kernel), "Fork vs Thread" (https://medium.com/obscure-system/fork-vs-thread-38e09ec099e2), and "Linux process and thread" (https://zliu.org/post/linux-process-and-thread) ("This means that to create a normal process fork() is used that further calls clone() with appropriate arguments while to create a thread or LWP, a function from pthread library calls clone() with relvant flags. So, the main difference is generated by using different flags that can be passed to clone() funciton(to be exact, it is a system call").
You may be confused by the fact that threads are called light-weight processes.
Or maybe I'm confused :)
If you have other information, please let me know. Thanks.
Jen
Dec 5, 2021, 18:08 by hjp-p...@hjp.at:
> On 2021-12-06 00:51:13 +0100, Jen Kris via Python-list wrote:
>
process and therefore also a new thread." From the Linux man pages https://www.man7.org/linux/man-pages/man2/fork.2.html, "The child process is created with a single thread—the one that called fork()."
I have a one-core one-thread instance at Digital Ocean available running Ubuntu 18.04. I can fork and create a new process on it, but it doesn't create a new thread because it doesn't have one available.
You may also want to see "Forking vs Threading" (https://www.geekride.com/fork-forking-vs-threading-thread-linux-kernel), "Fork vs Thread" (https://medium.com/obscure-system/fork-vs-thread-38e09ec099e2), and "Linux process and thread" (https://zliu.org/post/linux-process-and-thread) ("This means that to create a normal process fork() is used that further calls clone() with appropriate arguments while to create a thread or LWP, a function from pthread library calls clone() with relvant flags. So, the main difference is generated by using different flags that can be passed to clone() funciton(to be exact, it is a system call").
You may be confused by the fact that threads are called light-weight processes.
Or maybe I'm confused :)
If you have other information, please let me know. Thanks.
Jen
Dec 5, 2021, 18:08 by hjp-p...@hjp.at:
> On 2021-12-06 00:51:13 +0100, Jen Kris via Python-list wrote:
>
>> The C program creates two threads (using pthreads), one for itself and
>> one for the child process. On creation, the second pthread is pointed
>> to a C program that calls fork-execv to run the Python program. That
>> way Python runs on a separate thread.
>>
>
>> one for the child process. On creation, the second pthread is pointed
>> to a C program that calls fork-execv to run the Python program. That
>> way Python runs on a separate thread.
>>
>
Chris Angelico
Dec 6, 2021, 1:28:17 PM12/6/21
to
On Tue, Dec 7, 2021 at 4:10 AM Jen Kris via Python-list
<pytho...@python.org> wrote:
>
> I can't find any support for your comment that "Fork creates a new
> process and therefore also a new thread." From the Linux man pages https://www.man7.org/linux/man-pages/man2/fork.2.html, "The child process is created with a single thread—the one that called fork()."
>
> I have a one-core one-thread instance at Digital Ocean available running Ubuntu 18.04. I can fork and create a new process on it, but it doesn't create a new thread because it doesn't have one available.
>
A CPU core is capable of running one or more threads *concurrently*,
<pytho...@python.org> wrote:
>
> I can't find any support for your comment that "Fork creates a new
> process and therefore also a new thread." From the Linux man pages https://www.man7.org/linux/man-pages/man2/fork.2.html, "The child process is created with a single thread—the one that called fork()."
>
> I have a one-core one-thread instance at Digital Ocean available running Ubuntu 18.04. I can fork and create a new process on it, but it doesn't create a new thread because it doesn't have one available.
>
and having multiple cores obviously multiplies that out. But for what
you're doing here, there's no material difference between threads
running concurrently and threads switching out between themselves
(other than performance, of course). From the operating system and
application perspectives, a "thread" is one runnable thread.
Every Unix process must contain a minimum of one thread. As a rough
rule of thumb, a process owns resources, a thread runs code. Without a
thread, you can't run any code.
Forking a process creates a child process and leaves the parent
running. The child process must by definition have a minimum of one
thread, and the man page is stating that one thread is all it gets.
Hope that helps explain things a bit. I don't know exactly what's
going on in your code, but maybe that'll clarify the word "thread".
ChrisA
Barry Scott
Dec 6, 2021, 1:37:24 PM12/6/21
to
> On 6 Dec 2021, at 17:09, Jen Kris via Python-list <pytho...@python.org> wrote:
>
> I can't find any support for your comment that "Fork creates a new
> process and therefore also a new thread." From the Linux man pages https://www.man7.org/linux/man-pages/man2/fork.2.html, "The child process is created with a single thread—the one that called fork()."
All new processes on unix (may all OS) only ever have one thread when they start.
The thread-id of the first thread is the same as the process-id and referred to as the main thread.
>
> I have a one-core one-thread instance at Digital Ocean available running Ubuntu 18.04. I can fork and create a new process on it, but it doesn't create a new thread because it doesn't have one available.
It has one hardware core that support one hardware thread.
Linux can create as many software threads as it likes.
> You may also want to see "Forking vs Threading" (https://www.geekride.com/fork-forking-vs-threading-thread-linux-kernel), "Fork vs Thread" (https://medium.com/obscure-system/fork-vs-thread-38e09ec099e2), and "Linux process and thread" (https://zliu.org/post/linux-process-and-thread) ("This means that to create a normal process fork() is used that further calls clone() with appropriate arguments while to create a thread or LWP, a function from pthread library calls clone() with relvant flags. So, the main difference is generated by using different flags that can be passed to clone() funciton(to be exact, it is a system call").
>
> You may be confused by the fact that threads are called light-weight processes.
>
> Or maybe I'm confused :)
>
> If you have other information, please let me know. Thanks.
Barry
>
> Jen
>
>
> Dec 5, 2021, 18:08 by hjp-p...@hjp.at:
>
>> On 2021-12-06 00:51:13 +0100, Jen Kris via Python-list wrote:
>>
>>> The C program creates two threads (using pthreads), one for itself and
>>> one for the child process. On creation, the second pthread is pointed
>>> to a C program that calls fork-execv to run the Python program. That
>>> way Python runs on a separate thread.
>>>
>>
>> I think you have the relationship between processes and threads
>> process and therefore also a new thread.
>>
>>
>> hp
>>
>> --
>> _ | Peter J. Holzer | Story must make more sense than reality.
>> |_|_) | |
>> | | | h...@hjp.at | -- Charles Stross, "Creative writing
>> __/ | http://www.hjp.at/ | challenge!"
>>
>
> --
>>
>> --
>> _ | Peter J. Holzer | Story must make more sense than reality.
>> |_|_) | |
>> | | | h...@hjp.at | -- Charles Stross, "Creative writing
>> __/ | http://www.hjp.at/ | challenge!"
>>
>
> https://mail.python.org/mailman/listinfo/python-list
Barry Scott
Dec 6, 2021, 4:59:56 PM12/6/21
to
> On 6 Dec 2021, at 21:05, Jen Kris <jen...@tutanota.com> wrote:
>
> Here is what I don't understand from what you said. "The child process is created with a single thread—the one that called fork()." To me that implies that the thread that called fork() is the same thread as the child process. I guess you're talking about the distinction between logical threads and physical threads.
The thread that called fork is cloned into a new thread in the new process.
It has a clone of the processor registers of the thread, stack memory segment of that thread,
which means that it has all the stack variables and stack frames from the parent process's thread.
> But the main issue is your suggestion that I should call fork-execv from the thread that runs the main C program, not from a separate physical pthread. That would certainly eliminate the overhead of creating a new pthread.
Forget about physical threads, they only matter when you are performance tuning your code.
> I am working now to finish this, and I will try your suggestion of calling fork-execv from the "main" thread. When I reply back next I can give you a complete picture of what I'm doing.
>
> Your comments, and those of Peter Holzer and Chris Angelico, are most appreciated.
Barry
>
>
> Dec 6, 2021, 10:37 by ba...@barrys-emacs.org:
Jen Kris
Dec 6, 2021, 8:15:13 PM12/6/21
to
Here is what I don't understand from what you said. "The child process is created with a single thread—the one that called fork()." To me that implies that the thread that called fork() is the same thread as the child process. I guess you're talking about the distinction between logical threads and physical threads.
But the main issue is your suggestion that I should call fork-execv from the thread that runs the main C program, not from a separate physical pthread. That would certainly eliminate the overhead of creating a new pthread.
I am working now to finish this, and I will try your suggestion of calling fork-execv from the "main" thread. When I reply back next I can give you a complete picture of what I'm doing.
Your comments, and those of Peter Holzer and Chris Angelico, are most appreciated.
Your comments, and those of Peter Holzer and Chris Angelico, are most appreciated.
Jen Kris
Dec 8, 2021, 12:12:15 PM12/8/21
to
I started this post on November 29, and there have been helpful comments since then from Barry Scott, Cameron Simpson, Peter Holzer and Chris Angelico. Thanks to all of you.
I've found a solution that works for my purpose, and I said earlier that I would post the solution I found. If anyone has a better solution I would appreciate any feedback.
To recap, I'm using a pair of named pipes for IPC between C and Python. Python runs as a child process after fork-execv. The Python program continues to run concurrently in a while True loop, and responds to requests from C at intervals, and continues to run until it receives a signal from C to exit. C sends signals to Python, then waits to receive data back from Python. My problem was that C was blocked when Python started.
The solution was twofold: (1) for Python to run concurrently it must be a multiprocessing loop (from the multiprocessing module), and (2) Python must terminate its write strings with \n, or read will block in C waiting for something that never comes. The multiprocessing module sidesteps the GIL; without multiprocessing the GIL will block all other threads once Python starts.
Originally I used epoll() on the pipes. Cameron Smith and Barry Scott advised against epoll, and for this case they are right. Blocking pipes work here, and epoll is too much overhead for watching on a single file descriptor.
This is the Python code now:
#!/usr/bin/python3
from multiprocessing import Process
import os
print("Python is running")
child_pid = os.getpid()
print('child process id:', child_pid)
def f(a, b):
print("Python now in function f")
pr = os.open('/tmp/Pipe_01', os.O_RDONLY)
print("File Descriptor1 Opened " + str(pr))
pw = os.open('/tmp/Pipe_02', os.O_WRONLY)
print("File Descriptor2 Opened " + str(pw))
while True:
v = os.read(pr,64)
print("Python read from pipe pr")
print(v)
if v == b'99':
os.close(pr)
os.close(pw)
print("Python is terminating")
os._exit(os.EX_OK)
if v != "Send child PID":
os.write(pw, b"OK message received\n")
print("Python wrote back")
if __name__ == '__main__':
a = 0
b = 0
p = Process(target=f, args=(a, b,))
p.start()
p.join()
The variables a and b are not currently used in the body, but they will be later.
This is the part of the C code that communicates with Python:
fifo_fd1 = open(fifo_path1, O_WRONLY);
fifo_fd2 = open(fifo_path2, O_RDONLY);
status_write = write(fifo_fd1, py_msg_01, sizeof(py_msg_01));
if (status_write < 0) perror("write");
status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_01));
if (status_read < 0) perror("read");
printf("C received message 1 from Python\n");
printf("%.*s",(int)buf_len, fifo_readbuf);
status_write = write(fifo_fd1, py_msg_02, sizeof(py_msg_02));
if (status_write < 0) perror("write");
status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_02));
if (status_read < 0) perror("read");
printf("C received message 2 from Python\n");
printf("%.*s",(int)buf_len, fifo_readbuf);
// Terminate Python multiprocessing
printf("C is sending exit message to Python\n");
status_write = write(fifo_fd1, py_msg_03, 2);
printf("C is closing\n");
close(fifo_fd1);
close(fifo_fd2);
Screen output:
Python is running
child process id: 5353
Python now in function f
File Descriptor1 Opened 6
Thread created 0
File Descriptor2 Opened 7
Process ID: 5351
Parent Process ID: 5351
I am the parent
Core joined 0
I am the child
Python read from pipe pr
b'Hello to Python from C\x00\x00'
Python wrote back
C received message 1 from Python
OK message received
Python read from pipe pr
b'Message to Python 2\x00\x00'
Python wrote back
C received message 2 from Python
OK message received
C is sending exit message to Python
C is closing
Python read from pipe pr
b'99'
Python is terminating
Python runs on a separate thread (created with pthreads) because I want the flexibility of using this same basic code as a stand-alone .exe, or for a C extension from Python called with ctypes. If I use it as a C extension then I want the Python code on a separate thread because I can't have two instances of the Python interpreter running on one thread, and one instance will already be running on the main thread, albeit "suspended" by the call from ctypes.
So that's my solution: (1) Python multiprocessing module; (2) Python strings written to the pipe must be terminated with \n.
Thanks again to all who commented.
Dec 6, 2021, 13:33 by ba...@barrys-emacs.org:
>
>
>
I've found a solution that works for my purpose, and I said earlier that I would post the solution I found. If anyone has a better solution I would appreciate any feedback.
To recap, I'm using a pair of named pipes for IPC between C and Python. Python runs as a child process after fork-execv. The Python program continues to run concurrently in a while True loop, and responds to requests from C at intervals, and continues to run until it receives a signal from C to exit. C sends signals to Python, then waits to receive data back from Python. My problem was that C was blocked when Python started.
The solution was twofold: (1) for Python to run concurrently it must be a multiprocessing loop (from the multiprocessing module), and (2) Python must terminate its write strings with \n, or read will block in C waiting for something that never comes. The multiprocessing module sidesteps the GIL; without multiprocessing the GIL will block all other threads once Python starts.
Originally I used epoll() on the pipes. Cameron Smith and Barry Scott advised against epoll, and for this case they are right. Blocking pipes work here, and epoll is too much overhead for watching on a single file descriptor.
This is the Python code now:
#!/usr/bin/python3
from multiprocessing import Process
import os
print("Python is running")
child_pid = os.getpid()
print('child process id:', child_pid)
def f(a, b):
print("Python now in function f")
pr = os.open('/tmp/Pipe_01', os.O_RDONLY)
print("File Descriptor1 Opened " + str(pr))
pw = os.open('/tmp/Pipe_02', os.O_WRONLY)
print("File Descriptor2 Opened " + str(pw))
while True:
v = os.read(pr,64)
print("Python read from pipe pr")
print(v)
if v == b'99':
os.close(pr)
os.close(pw)
print("Python is terminating")
os._exit(os.EX_OK)
if v != "Send child PID":
os.write(pw, b"OK message received\n")
print("Python wrote back")
if __name__ == '__main__':
a = 0
b = 0
p = Process(target=f, args=(a, b,))
p.start()
p.join()
The variables a and b are not currently used in the body, but they will be later.
This is the part of the C code that communicates with Python:
fifo_fd1 = open(fifo_path1, O_WRONLY);
fifo_fd2 = open(fifo_path2, O_RDONLY);
status_write = write(fifo_fd1, py_msg_01, sizeof(py_msg_01));
if (status_write < 0) perror("write");
status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_01));
if (status_read < 0) perror("read");
printf("C received message 1 from Python\n");
printf("%.*s",(int)buf_len, fifo_readbuf);
status_write = write(fifo_fd1, py_msg_02, sizeof(py_msg_02));
if (status_write < 0) perror("write");
status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_02));
if (status_read < 0) perror("read");
printf("C received message 2 from Python\n");
printf("%.*s",(int)buf_len, fifo_readbuf);
// Terminate Python multiprocessing
printf("C is sending exit message to Python\n");
status_write = write(fifo_fd1, py_msg_03, 2);
printf("C is closing\n");
close(fifo_fd1);
close(fifo_fd2);
Screen output:
Python is running
child process id: 5353
Python now in function f
File Descriptor1 Opened 6
Thread created 0
File Descriptor2 Opened 7
Process ID: 5351
Parent Process ID: 5351
I am the parent
Core joined 0
I am the child
Python read from pipe pr
b'Hello to Python from C\x00\x00'
Python wrote back
C received message 1 from Python
OK message received
Python read from pipe pr
b'Message to Python 2\x00\x00'
Python wrote back
C received message 2 from Python
OK message received
C is sending exit message to Python
C is closing
Python read from pipe pr
b'99'
Python is terminating
Python runs on a separate thread (created with pthreads) because I want the flexibility of using this same basic code as a stand-alone .exe, or for a C extension from Python called with ctypes. If I use it as a C extension then I want the Python code on a separate thread because I can't have two instances of the Python interpreter running on one thread, and one instance will already be running on the main thread, albeit "suspended" by the call from ctypes.
So that's my solution: (1) Python multiprocessing module; (2) Python strings written to the pipe must be terminated with \n.
Thanks again to all who commented.
Dec 6, 2021, 13:33 by ba...@barrys-emacs.org:
>> Here is what I don't understand from what you said. "The child process is created with a single thread—the one that called fork()." To me that implies that the thread that called fork() is the same thread as the child process. I guess you're talking about the distinction between logical threads and physical threads.
>>
>
>>
>
> The thread that called fork is cloned into a new thread in the new process.
> It has a clone of the processor registers of the thread, stack memory segment of that thread,
> which means that it has all the stack variables and stack frames from the parent process's thread.
>
>
>
> It has a clone of the processor registers of the thread, stack memory segment of that thread,
> which means that it has all the stack variables and stack frames from the parent process's thread.
>
>
>
>> But the main issue is your suggestion that I should call fork-execv from the thread that runs the main C program, not from a separate physical pthread. That would certainly eliminate the overhead of creating a new pthread.
>>
>
>>
>
> Forget about physical threads, they only matter when you are performance tuning your code.
>
>
>
>
>> I am working now to finish this, and I will try your suggestion of calling fork-execv from the "main" thread. When I reply back next I can give you a complete picture of what I'm doing.
>>
>> Your comments, and those of Peter Holzer and Chris Angelico, are most appreciated.
>>
>
> Barry
>>
>> Your comments, and those of Peter Holzer and Chris Angelico, are most appreciated.
>>
>
>
>
>
Cameron Simpson
Dec 8, 2021, 5:11:34 PM12/8/21
to
On 08Dec2021 18:11, Jen Kris <jen...@tutanota.com> wrote:
>Python must terminate its write strings with \n, or read will block in
>C waiting for something that never comes.
There are two aspects to this:
>Python must terminate its write strings with \n, or read will block in
>C waiting for something that never comes.
- if upstream is rding "lines of text" then you need a newline to
terminate the lines
- you (probably) should flush the output pipe (Python to C) after the
newline
I see you're using file descriptors and os.write() to sent data. This is
unbuffered, so there is nothing to flush, so you have not encountered
the second point above.
But if you shift to using a Python file object for your output (eg
f=os.fdopen(pw)), which would let you use print() or any number of other
things which do things with Python files) your file object would have a
buffer and normally that would not be sent to the pipe unless it was
full.
So your deadlock issue has 2 components:
- you need to terminate your records for upstream (C) to see complete
records. Your records are lines, so you need a newline character.
- you need to ensure the whole record has been sent upstream (been
written to the pipe). If you use a buffered Python file object for
your output, you need to flush it at your synchronisation points or
upstream will not receive the buffer contents. That synchronisation
point for you is the end of the record.
Hopefully this makes the flow considerations more clear.
Cheers,
Cameron Simpson <c...@cskk.id.au>
Peter J. Holzer
Dec 8, 2021, 5:36:44 PM12/8/21
to
On 2021-12-08 18:11:48 +0100, Jen Kris via Python-list wrote:
> To recap, I'm using a pair of named pipes for IPC between C and
> Python. Python runs as a child process after fork-execv. The Python
> program continues to run concurrently in a while True loop, and
> responds to requests from C at intervals, and continues to run until
> it receives a signal from C to exit. C sends signals to Python, then
> waits to receive data back from Python. My problem was that C was
> blocked when Python started.
>
> The solution was twofold: (1) for Python to run concurrently it must
> be a multiprocessing loop (from the multiprocessing module),
I don't see how this could achieve anything. It starts another (third)
> To recap, I'm using a pair of named pipes for IPC between C and
> Python. Python runs as a child process after fork-execv. The Python
> program continues to run concurrently in a while True loop, and
> responds to requests from C at intervals, and continues to run until
> it receives a signal from C to exit. C sends signals to Python, then
> waits to receive data back from Python. My problem was that C was
> blocked when Python started.
>
> The solution was twofold: (1) for Python to run concurrently it must
> be a multiprocessing loop (from the multiprocessing module),
process, but then it just does all the work in that process and just
waits for it. Doing the same work in the original Python process should
have exactly the same effect.
> and (2) Python must terminate its write strings with \n, or read will
> block in C waiting for something that never comes.
That's also strange. You are using os.write in Python and read in C,
both of which shoudn't care about newlines.
> The multiprocessing module sidesteps the GIL; without multiprocessing
> the GIL will block all other threads once Python starts.
There is absolutely no way the GIL could block threads in your C
program. And your Python code doesn't need to use more than one thread
or process.
Barry Scott
Dec 9, 2021, 6:57:35 AM12/9/21
to
> On 8 Dec 2021, at 17:11, Jen Kris <jen...@tutanota.com> wrote:
>
> I started this post on November 29, and there have been helpful comments since then from Barry Scott, Cameron Simpson, Peter Holzer and Chris Angelico. Thanks to all of you.
>
> I've found a solution that works for my purpose, and I said earlier that I would post the solution I found. If anyone has a better solution I would appreciate any feedback.
>
> To recap, I'm using a pair of named pipes for IPC between C and Python. Python runs as a child process after fork-execv. The Python program continues to run concurrently in a while True loop, and responds to requests from C at intervals, and continues to run until it receives a signal from C to exit. C sends signals to Python, then waits to receive data back from Python. My problem was that C was blocked when Python started.
>
> The solution was twofold: (1) for Python to run concurrently it must be a multiprocessing loop (from the multiprocessing module), and (2) Python must terminate its write strings with \n, or read will block in C waiting for something that never comes. The multiprocessing module sidesteps the GIL; without multiprocessing the GIL will block all other threads once Python starts.
>
>
> Originally I used epoll() on the pipes. Cameron Smith and Barry Scott advised against epoll, and for this case they are right. Blocking pipes work here, and epoll is too much overhead for watching on a single file descriptor.
>
> This is the Python code now:
>
> #!/usr/bin/python3
> from multiprocessing import Process
You already have feedback that multiprocessing is not required.
> Originally I used epoll() on the pipes. Cameron Smith and Barry Scott advised against epoll, and for this case they are right. Blocking pipes work here, and epoll is too much overhead for watching on a single file descriptor.
>
> This is the Python code now:
>
> #!/usr/bin/python3
> from multiprocessing import Process
> import os
>
> print("Python is running")
>
> child_pid = os.getpid()
> print('child process id:', child_pid)
>
> def f(a, b):
>
> print("Python now in function f")
>
> pr = os.open('/tmp/Pipe_01', os.O_RDONLY)
> print("File Descriptor1 Opened " + str(pr))
> pw = os.open('/tmp/Pipe_02', os.O_WRONLY)
> print("File Descriptor2 Opened " + str(pw))
>
> while True:
>
> v = os.read(pr,64)
> print("Python read from pipe pr")
> print(v)
>
> if v == b'99':
> os.close(pr)
> os.close(pw)
> print("Python is terminating")
> os._exit(os.EX_OK)
>
> if v != "Send child PID":
> os.write(pw, b"OK message received\n")
> print("Python wrote back")
>
> if __name__ == '__main__':
> a = 0
> b = 0
> p = Process(target=f, args=(a, b,))
> p.start()
> p.join()
>
> The variables a and b are not currently used in the body, but they will be later.
>
> This is the part of the C code that communicates with Python:
>
> fifo_fd1 = open(fifo_path1, O_WRONLY);
> fifo_fd2 = open(fifo_path2, O_RDONLY);
>
> status_write = write(fifo_fd1, py_msg_01, sizeof(py_msg_01));
> if (status_write < 0) perror("write");
>
> status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_01));
> if (status_read < 0) perror("read");
> printf("C received message 1 from Python\n");
> printf("%.*s",(int)buf_len, fifo_readbuf);
The length of the data read is in status_read not buf_len.
> status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_01));
> if (status_read < 0) perror("read");
> printf("C received message 1 from Python\n");
> printf("%.*s",(int)buf_len, fifo_readbuf);
>
> status_write = write(fifo_fd1, py_msg_02, sizeof(py_msg_02));
Is it a C string? If so you want to write sizeof(py_msg_02)-1 to avoid sending the trailing NUL (0)
of the C string.
> if (status_write < 0) perror("write");
>
> status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_02));
> if (status_read < 0) perror("read");
> printf("C received message 2 from Python\n");
> printf("%.*s",(int)buf_len, fifo_readbuf);
The length of the data read is in status_read not buf_len.
> status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_02));
> if (status_read < 0) perror("read");
> printf("C received message 2 from Python\n");
> printf("%.*s",(int)buf_len, fifo_readbuf);
At no point do I see in this C code the need for a \n in the data sent between python and C.
>
> // Terminate Python multiprocessing
> printf("C is sending exit message to Python\n");
> status_write = write(fifo_fd1, py_msg_03, 2);
If py_msg_03 only has the exit msg in it the use sizeof(py_msg_03).
If its a C string the subtract 1 for the trailing NUL (0).
>
> printf("C is closing\n");
> close(fifo_fd1);
> close(fifo_fd2);
>
> Screen output:
>
> Python is running
> child process id: 5353
> Python now in function f
> File Descriptor1 Opened 6
> Thread created 0
> File Descriptor2 Opened 7
> Process ID: 5351
> Parent Process ID: 5351
> I am the parent
> Core joined 0
> I am the child
> Python read from pipe pr
> b'Hello to Python from C\x00\x00'
> Python wrote back
> C received message 1 from Python
> OK message received
> Python read from pipe pr
> b'Message to Python 2\x00\x00'
0 new messages