Thread must sleep forever (Try to double-lock mutex?)

[Frederick Gotham]

I'm working on an embedded Linux project.

When the computer boots up, a program called "Supervisor" is used to make sure that all background processes get immediately restarted if they crash.

So let's say a process called "gas_monitor" gets run at startup. If it crashes 3 days later, then Supervisor will immediately restart it.

Supervisor has limits though: If it tries to restart a process and it immediately crashes again, then it will try one more time to restart it, and then it will stop trying.

When the computer is configured a particular way, for example let's say we change the config file from "gas_source=alkane" to "gas_source=halogen", then the "gas_monitor" program no longer has a purpose.

Since "gas_monitor" is no longer needed, I can close it. . . but then Supervisor will try to restart it two more times, and it will close another two times. Then if the config file is changed back to "alkane", Supervisor won't try to restart the gas_monitor process because it's already failed the maximum times of two.

So when 'gas_monitor' isn't needed, I just need to put it to sleep. I want to do this the most efficient way possible (as I'm running embedded Linux and so CPU cycles are to be conserved).

Using C++11, I think the best way to do this is:

std::mutex m;
m.lock();
m.lock();

Am I right?

[Paavo Helde]

Nope, this is about the worst way to do that, if it appears to work then
it's accidental. Double-locking of a std::mutex in the same thread is
forbidden. From https://en.cppreference.com/w/cpp/thread/mutex :

"A calling thread must not own the mutex prior to calling lock or try_lock."

Also, when in infinite sleep your program won't wake up and continue to
work as expected, when the config file changes again.

Instead, what you can do is to set up an inotify disk monitor thread
which monitors the config file and notifies the main thread when the
config file changes. The main thread would wait in a
std::condition_variable::wait(); when notified, it would reread the
config file and decide if it should start running again or continue to
wait. I have no idea if your embedded Linux supports inotify.

Alternatively, you can set up another thread which handles the SIGHUP
signal via sigwait() and which notifies the main thread when the process
is sent a SIGHUP. The main thread will again wait in a
std::condition_variable::wait() meanwhile. This means that after
changing the config files one has to send the SIGHUP signal to the
process so it can reread the config and continue running or wait as desired.

[Ian Collins]

On 22/10/2019 22:09, Frederick Gotham wrote:
> I'm working on an embedded Linux project.
>
> When the computer boots up, a program called "Supervisor" is used to
> make sure that all background processes get immediately restarted if
> they crash.
>
> So let's say a process called "gas_monitor" gets run at startup. If
> it crashes 3 days later, then Supervisor will immediately restart
> it.

Why don't you just run your processes as services?

> Using C++11, I think the best way to do this is:
>
> std::mutex m; m.lock(); m.lock();
>
> Am I right?

No! You can't re-lock a locked mutex.

--
Ian.

[melzzzzz]

Ian Collins <ian-...@hotmail.com> Wrote in message:r
> On 22/10/2019 22:09, Frederick Gotham wrote:> I'm working on an embedded Linux project.> > When the computer boots up, a program called "Supervisor" is used to> make sure that all background processes get immediately restarted if> they crash.> > So let's say a process called "gas_monitor" gets run at startup. If> it crashes 3 days later, then Supervisor will immediately restart> it.Why don't you just run your processes as services?> Using C++11, I think the best way to do this is:> > std::mutex m; m.lock(); m.lock();> > Am I right?No! You can't re-lock a locked mutex.

is there posibillity to specify recursive mutex? i recall there is
one?


--
Press any key to continue or any other to quit....

[Paavo Helde]

Yes there are recursive mutexes (which basically just encourage sloppy
coding).

However, OP wanted a deadlock (a very bad idea but that's what he
wanted), but a recursive mutex would not block if locked twice in the
same thread.

[melzzzzz]

Paavo Helde <myfir...@osa.pri.ee> Wrote in message:r

"However, OP wanted a deadlock"

Strange, deadlock is a bug :)

[Chris Vine]

If you are writing multi-threaded embedded software professionally then
that is a somewhat alarming question.

The answer is no. You might get the deadlock you seem to want, or you
might not. The C++ standard doesn't specify what will happen, as double
locking a non-recursive mutex is a programming error. If you do get the
deadlock you seem to want then there is no way legal way of awakening
the thread should your configuration change (only the thread which
locks a mutex can properly unlock it).

Look up C++ condition variables, which would do what you want. Note
that these can spuriously awake, so put the check of the condition upon
which the wait occurs (which could just be a boolean value) in a while
loop.

[Scott Lurndal]

Frederick Gotham <cauldwel...@gmail.com> writes:
>I'm working on an embedded Linux project.
>

>When the computer boots up, a program called "Supervisor" is used to make s=

>ure that all background processes get immediately restarted if they crash.

In other words, you're recreating the almost 50 year old "init" daemon.

[Scott Lurndal]

Frederick Gotham <cauldwel...@gmail.com> writes:
>I'm working on an embedded Linux project.

>So when 'gas_monitor' isn't needed, I just need to put it to sleep. I want =
>to do this the most efficient way possible (as I'm running embedded Linux a=

>nd so CPU cycles are to be conserved).

kill( pid_of_gas_monitor, SIGSTOP );

when you're ready to continue the process:

kill (pid_of_gas_monitor, SIGCONT );

[Frederick Gotham]

On Tuesday, October 22, 2019 at 1:30:30 PM UTC+1, Chris Vine wrote:

> If you do get the
> deadlock you seem to want then there is no way legal way of awakening
> the thread should your configuration change (only the thread which
> locks a mutex can properly unlock it).

Everything gets killed and restarted when the config changes.

[Frederick Gotham]

On Tuesday, October 22, 2019 at 2:07:37 PM UTC+1, Scott Lurndal wrote:

> Frederick Gotham writes:
> >I'm working on an embedded Linux project.
> >
> >When the computer boots up, a program called "Supervisor" is used to make s=
> >ure that all background processes get immediately restarted if they crash.
>
> In other words, you're recreating the almost 50 year old "init" daemon.

No I think the guy in charge of that part is using the stuff from supervisord.org

[Chris Vine]

To do that reliably on a mutex which has in fact deadlocked, you would
need to kill the entire process within which that particular blocking
thread is running.

Anyway, as you have stated that you are using linux don't do this as you
are relying on the default behaviour provided from time to time by
g++'s C++ implementation when calling up the native glibc pthreads
implementation. The behaviour of pthread_mutex_lock() on relocking for
a PTHREAD_MUTEX_DEFAULT mutex is stated by the SUS as 'undefined' (it
may be but does not have to be the behaviour of any one of the other
three types below - instead for efficiency reasons it could just crash
your program). When explicitly constructed as of type
PTHREAD_MUTEX_ERRORCHECK it is defined to return with an error. When
explicitly constructed as of type PTHREAD_MUTEX_NORMAL it will
deadlock. When explicitly constructed as of type
PTHREAD_MUTEX_RECURSIVE it will lock recursively and then return with
no error. See this:
https://linux.die.net/man/3/pthread_mutexattr_settype

If you don't want to use condition variables and you never want to
re-awaken the thread in question then you can use select() with empty
sets and NULL timeout, or sleep or nanosleep with a very long timeout.
To account for signals you would need to put these in a while loop.

[Jorgen Grahn]

On Tue, 2019-10-22, Frederick Gotham wrote:
> I'm working on an embedded Linux project.
>
> When the computer boots up, a program called "Supervisor" is used to
> make sure that all background processes get immediately restarted if
> they crash.
>
> So let's say a process called "gas_monitor" gets run at startup. If
> it crashes 3 days later, then Supervisor will immediately restart
> it.
>
> Supervisor has limits though: If it tries to restart a process and
> it immediately crashes again, then it will try one more time to
> restart it, and then it will stop trying.

> When the computer is configured a particular way, for example let's
> say we change the config file from "gas_source=alkane" to
> "gas_source=halogen", then the "gas_monitor" program no longer has a
> purpose.

I'd prefer software documented as "if you change the config, you have
to restart" if at all possible. It's the traditional Unix way (well,
except daemons often reload their config on SIGHUP, but I bet many do that
by releasing resources and exec()ing themselves).

> Since "gas_monitor" is no longer needed, I can close it. . . but
> then Supervisor will try to restart it two more times, and it will
> close another two times. Then if the config file is changed back to
> "alkane", Supervisor won't try to restart the gas_monitor process
> because it's already failed the maximum times of two.
>
> So when 'gas_monitor' isn't needed, I just need to put it to
> sleep.

It's the wrong design. If it's important to not run it, then this
Supervisor thing needs to know (based on config) what processes to
supervise, and stop/never start the ones that are not needed.

Things get complicated once you have a Supervisor supervising things
which /pretend/ to work, when everyone knows they are really asleep on
the job.

> I want to do this the most efficient way possible (as I'm
> running embedded Linux and so CPU cycles are to be conserved).

What are you going to use those cycles for instead? Shouldn't you be
waiting for new config? Surely waiting on I/O doesn't use up CPU
cycles.

(Nothing C++-specific about the things above, by the way.)

> Using C++11, I think the best way to do this is:
>
> std::mutex m;
> m.lock();
> m.lock();
>
> Am I right?

I'd prefer pause(3) or similar, if you really never want to wake up.
It's not C++, but it seems unwise to avoid Unix APIs in a project like
this.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

[Chris M. Thomasson]

On 10/22/2019 2:09 AM, Frederick Gotham wrote:
> I'm working on an embedded Linux project.
>
> When the computer boots up, a program called "Supervisor" is used to make sure that all background processes get immediately restarted if they crash.
>
> So let's say a process called "gas_monitor" gets run at startup. If it crashes 3 days later, then Supervisor will immediately restart it.
>
> Supervisor has limits though: If it tries to restart a process and it immediately crashes again, then it will try one more time to restart it, and then it will stop trying.
>
> When the computer is configured a particular way, for example let's say we change the config file from "gas_source=alkane" to "gas_source=halogen", then the "gas_monitor" program no longer has a purpose.
>
> Since "gas_monitor" is no longer needed, I can close it. . . but then Supervisor will try to restart it two more times, and it will close another two times. Then if the config file is changed back to "alkane", Supervisor won't try to restart the gas_monitor process because it's already failed the maximum times of two.

Send a request to the Supervisor to close the "gas_monitor"?

In other words, the Supervisor should have a shared memory queue, or
even a socket such that other programs can communicate with it. To shut
down the gas_monitor, have the configuration program ask the Supervisor
to shut it down. To start it back up, just tell the Supervisor to go
ahead and do that.

> So when 'gas_monitor' isn't needed, I just need to put it to sleep. I want to do this the most efficient way possible (as I'm running embedded Linux and so CPU cycles are to be conserved).
>
> Using C++11, I think the best way to do this is:
>
> std::mutex m;
> m.lock();
> m.lock();
>
> Am I right?
>

This is bad mojo. Are you familiar with robust mutexs?

[Frederick Gotham]

On Tuesday, October 22, 2019 at 9:10:24 PM UTC+1, Jorgen Grahn wrote:

> Things get complicated once you have a Supervisor supervising things
> which /pretend/ to work, when everyone knows they are really asleep on
> the job.

I don't want to complicate the Supervisor, and so I'm happy to put the gas_monitor process to sleep.

[Chris M. Thomasson]

But, won't the Supervisor just try to restart it? Locking a
non-recursive mutex twice in the same thread is an error. This can be
solved multiple ways. The easy way is to create the Supervisor as a
service/ daemon. Robust mutexs can be used to detect when a process dies.

[Jorgen Grahn]

The software (as I understand it) is Unix-specific; standard practice
there is to have parent--child process relationships, and to use the
wait() family of functions to detect when a child dies.

[Jorgen Grahn]

In a way you /are/ complicating it already, by stretching the meaning
of "supervise". Especially if there is some kind of heartbeat
mechanism too, but maybe there isn't.

If you /really/ want to sleep forever and consume minimal resources on
Unix, the optimal way would be to close all file descriptors, release
anything else which would survive an exec(), and exec() a tiny
'sleep_forever' binary.

[Scott Lurndal]

Or poll periodically using kill(pid, 0); kill will return ESRCH if
the process doesn't exist.

Using SIGSTOP to suspend a process (either in-context using raise(3)
or externally from a monitor process using kill(2)) is the most efficient
way of pausing a process.

[Chris M. Thomasson]

That's fine. Missed the Unix part. Imvho, it would be nice for other
processes to be able to ask the Supervisor to shutdown/pause/whatever
one of its children. Creating a little webserver per main component of a
system can work pretty good. Being able to control the Supervisor from a
browser can be nice.

[red floyd]

Or, close all fds, set all signals to SIG_IGN, and run the following
loop:

for (;;) pause();

[Ian Collins]

Or simply use the platform's service management framework rather than
reinventing it!

--
Ian.

[Scott Lurndal]

raise(SIGSTOP);

[Jorgen Grahn]

On Fri, 2019-10-25, Scott Lurndal wrote:
> red floyd <no....@its.invalid> writes:
>>On 10/24/19 1:46 AM, Jorgen Grahn wrote:

...

>>> If you /really/ want to sleep forever and consume minimal resources on
>>> Unix, the optimal way would be to close all file descriptors, release
>>> anything else which would survive an exec(), and exec() a tiny
>>> 'sleep_forever' binary.
>>
>>Or, close all fds, set all signals to SIG_IGN, and run the following
>>loop:
>>
>> for (;;) pause();
>
> raise(SIGSTOP);

The exec() solution has one more benefit: it frees up virtual memory.
But we're pretty far offtopic now; I brought it up as a reminder that
platform-specific features are sometimes the best tool for the job.

[Scott Lurndal]

Jorgen Grahn <grahn...@snipabacken.se> writes:
>On Fri, 2019-10-25, Scott Lurndal wrote:
>> red floyd <no....@its.invalid> writes:
>>>On 10/24/19 1:46 AM, Jorgen Grahn wrote:
>...
>>>> If you /really/ want to sleep forever and consume minimal resources on
>>>> Unix, the optimal way would be to close all file descriptors, release
>>>> anything else which would survive an exec(), and exec() a tiny
>>>> 'sleep_forever' binary.
>>>
>>>Or, close all fds, set all signals to SIG_IGN, and run the following
>>>loop:
>>>
>>> for (;;) pause();
>>
>> raise(SIGSTOP);
>
>The exec() solution has one more benefit: it frees up virtual memory.

Although the only overhead associated with virtual memory is the
page tables. The OS can always reclaim the physical memory by
swapping out dirty pages and replacing clean pages owned by the
SIGSTOP'd process.

[Queequeg]

Frederick Gotham <cauldwel...@gmail.com> wrote:

> Everything gets killed and restarted when the config changes.

If you *really* must go with this path and can't change it to be written
properly (as others suggested), I'd suggest:

for (;;) pause();

--
https://www.youtube.com/watch?v=9lSzL1DqQn0

[Queequeg]

Scott Lurndal <sc...@slp53.sl.home> wrote:

> Although the only overhead associated with virtual memory is the
> page tables. The OS can always reclaim the physical memory by
> swapping out dirty pages and replacing clean pages owned by the
> SIGSTOP'd process.

If there's swap. It's not obvious on an embedded platform.

And I agree, raise(SIGSTOP) would be better than looping on pause(). But
will the process be stopped before raise() returns? My test shows that
yes, but I don't know if it's guaranteed or only a coincidence.

--
https://www.youtube.com/watch?v=9lSzL1DqQn0

[Scott Lurndal]

Yes, it is guaranteed.

[Chris M. Thomasson]

That works. There is usually a basic start/resume/pause/shutdown
protocol wrt services. I am just fond of creating a little webserver for
each main service.

[Queequeg]

Scott Lurndal <sc...@slp53.sl.home> wrote:

>>And I agree, raise(SIGSTOP) would be better than looping on pause(). But
>>will the process be stopped before raise() returns? My test shows that
>>yes, but I don't know if it's guaranteed or only a coincidence.
>
> Yes, it is guaranteed.

Ok, thanks.

--
https://www.youtube.com/watch?v=9lSzL1DqQn0