Importing djb's public domain daemontools?

21 views
Skip to first unread message

Jos Backus

unread,
Jan 16, 2012, 3:53:00 PM1/16/12
to freebs...@freebsd.org
http://cr.yp.to/distributors.html says:

"What are the distribution terms for daemontools?
2007.12.28: I hereby place the daemontools package (in particular,
daemontools-0.76.tar.gz, with MD5 checksum
1871af2453d6e464034968a0fbcb2bfc) into the public domain. The package is no
longer copyrighted."

FreeBSD could use a lightweight, robust service management framework so we
can finally get rid of the messy and unreliable pidfile concept. UNIX
already provides a much more elegant solution using fork()/exec()/wait() in
combination with the process table. daemontools provides a solid,
time-tested implementation of said mechanism.

There's also an updated version of daemontools by Bruce Guenter at
http://untroubled.org/daemontools-encore/ which among other improvements
has support for finish script functionality, which is very useful for
alerting/monitoring of service crashes. Its LICENSE file suggests that it
is released under a BSD-like license.

Thoughts?

--
Jos Backus
jos at catnook.com
_______________________________________________
freebs...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch...@freebsd.org"

Doug Barton

unread,
Jan 16, 2012, 9:53:05 PM1/16/12
to Jos Backus, freebs...@freebsd.org
On 01/16/2012 12:53, Jos Backus wrote:
>
> Thoughts?

This is already available in ports.

--

It's always a long day; 86400 doesn't fit into a short.

Breadth of IT experience, and depth of knowledge in the DNS.
Yours for the right price. :) http://SupersetSolutions.com/

Jos Backus

unread,
Jan 16, 2012, 10:41:15 PM1/16/12
to Doug Barton, freebs...@freebsd.org
On Jan 16, 2012 6:53 PM, "Doug Barton" <do...@freebsd.org> wrote:
>
> On 01/16/2012 12:53, Jos Backus wrote:
> >
> > Thoughts?
>
> This is already available in ports.

I realize that.

If FreeBSD had a solid solution out of the box, all this pidfile hackery in
the base system wouldn't be necessary. I always thought FreeBSD was about
good engineering. Perpetuating the pidfile mess in the base is not a sign
of good engineering.

Jos

Doug Barton

unread,
Jan 17, 2012, 12:10:37 AM1/17/12
to Jos Backus, freebs...@freebsd.org
On 01/16/2012 19:41, Jos Backus wrote:
> On Jan 16, 2012 6:53 PM, "Doug Barton" <do...@freebsd.org> wrote:
>>
>> On 01/16/2012 12:53, Jos Backus wrote:
>>>
>>> Thoughts?
>>
>> This is already available in ports.
>
> I realize that.

Good, then we're done. :)

> If FreeBSD had a solid solution out of the box, all this pidfile hackery in
> the base system wouldn't be necessary.

We don't do religious wars here. We especially don't do trollbait from
djb acolytes. The "pidfile hackery" that we currently have works just
fine in the vast majority of cases. The fact that it doesn't meet some
people's ideas of architectural purity is totally beside the point.

> I always thought FreeBSD was about
> good engineering. Perpetuating the pidfile mess in the base is not a sign
> of good engineering.

FreeBSD is about giving people choices. Those who want to use
daemontools can do that.

And lest people think that I'm just hating on daemontools, I'm not. I
use it for some things. But converting everything in the base to use it
is a non-starter, even if we wanted to import it, which I don't see any
need to do.


Doug

Jos Backus

unread,
Jan 17, 2012, 1:32:02 AM1/17/12
to Doug Barton, freebs...@freebsd.org
Hi Doug,

On Mon, Jan 16, 2012 at 9:10 PM, Doug Barton <do...@freebsd.org> wrote:

> On 01/16/2012 19:41, Jos Backus wrote:
> > On Jan 16, 2012 6:53 PM, "Doug Barton" <do...@freebsd.org> wrote:
> >>
> >> On 01/16/2012 12:53, Jos Backus wrote:
> >>>
> >>> Thoughts?
> >>
> >> This is already available in ports.
> >
> > I realize that.
>
> Good, then we're done. :)
>

Heh :)


>
> > If FreeBSD had a solid solution out of the box, all this pidfile hackery
> in
> > the base system wouldn't be necessary.
>
> We don't do religious wars here. We especially don't do trollbait from
> djb acolytes. The "pidfile hackery" that we currently have works just
> fine in the vast majority of cases. The fact that it doesn't meet some
> people's ideas of architectural purity is totally beside the point.
>
>

I want/need a solution that works in (nearly) all cases and is devoid of
complex code trying to track state that is already represented elsewhere in
the system (the process table and the parent/child process relationship). I
want a solution that can reliably handle a crashing server that doesn't
clean up its pidfile (the finish script functionality in daemontools-encore
provides this), and I want a unified control interface for the services
running on a box, a la launchd or what have you. This isn't about religion
but about missing base system functionality - the ability to reliably
control services running on a box.

> > I always thought FreeBSD was about
> > good engineering. Perpetuating the pidfile mess in the base is not a sign
> > of good engineering.
>
> FreeBSD is about giving people choices. Those who want to use
> daemontools can do that.
>

I thought the motto was "tools, not policies" ;)


>
> And lest people think that I'm just hating on daemontools, I'm not. I
> use it for some things. But converting everything in the base to use it
> is a non-starter, even if we wanted to import it, which I don't see any
> need to do.
>

Straw man. I'm asking for FreeBSD to *support* this functionality out of
the box, just like OS X, Solaris, AIX and some Linux versions (with
systemd). The closest FreeBSD has today is init, and it's not flexible
enough to do the job.

ISTR the last time I mentioned this, the debate was all about the code's
license, not about the functionality. I'm glad we at least now can discuss
whether this functionality belongs in the base system, as the license
should no longer be an issue (it's public domain).

Jos

--
Jos Backus
jos at catnook.com

Warner Losh

unread,
Jan 17, 2012, 2:10:30 AM1/17/12
to Doug Barton, freebs...@freebsd.org

On Jan 16, 2012, at 10:10 PM, Doug Barton wrote:

> On 01/16/2012 19:41, Jos Backus wrote:
>> On Jan 16, 2012 6:53 PM, "Doug Barton" <do...@freebsd.org> wrote:
>>>
>>> On 01/16/2012 12:53, Jos Backus wrote:
>>>>
>>>> Thoughts?
>>>
>>> This is already available in ports.
>>
>> I realize that.
>
> Good, then we're done. :)

Not necessarily...

>> If FreeBSD had a solid solution out of the box, all this pidfile hackery in
>> the base system wouldn't be necessary.
>
> We don't do religious wars here. We especially don't do trollbait from
> djb acolytes. The "pidfile hackery" that we currently have works just
> fine in the vast majority of cases. The fact that it doesn't meet some
> people's ideas of architectural purity is totally beside the point.

This isn't a religious war. This is someone coming to us and saying that it might be a good idea to clean up the mess by importing a tiny bit of extra code into the base. Seems like how we've always done things :)

>> I always thought FreeBSD was about
>> good engineering. Perpetuating the pidfile mess in the base is not a sign
>> of good engineering.
>
> FreeBSD is about giving people choices. Those who want to use
> daemontools can do that.
>
> And lest people think that I'm just hating on daemontools, I'm not. I
> use it for some things. But converting everything in the base to use it
> is a non-starter, even if we wanted to import it, which I don't see any
> need to do.

I'm not convinced it is a non-starter. I'd fully support Jos if he wanted to commit the code and had done the leg work to do it. I wouldn't support just importing the daemontools and leaving it at that. If that's the plan, then leaving it in ports is the best bet.

Let's not dismiss this out of hand.

Warner_______________________________________________

Doug Barton

unread,
Jan 17, 2012, 2:34:13 AM1/17/12
to Warner Losh, freebs...@freebsd.org
On 01/16/2012 23:10, Warner Losh wrote:
>
> On Jan 16, 2012, at 10:10 PM, Doug Barton wrote:
>
>> On 01/16/2012 19:41, Jos Backus wrote:
>>> On Jan 16, 2012 6:53 PM, "Doug Barton" <do...@freebsd.org>
>>> wrote:
>>>>
>>>> On 01/16/2012 12:53, Jos Backus wrote:
>>>>>
>>>>> Thoughts?
>>>>
>>>> This is already available in ports.
>>>
>>> I realize that.
>>
>> Good, then we're done. :)
>
> Not necessarily...
>
>>> If FreeBSD had a solid solution out of the box, all this pidfile
>>> hackery in the base system wouldn't be necessary.
>>
>> We don't do religious wars here. We especially don't do trollbait
>> from djb acolytes. The "pidfile hackery" that we currently have
>> works just fine in the vast majority of cases. The fact that it
>> doesn't meet some people's ideas of architectural purity is totally
>> beside the point.
>
> This isn't a religious war.

You obviously haven't spent a lot of time dealing with djb'ites. Your
warning sign should have been "messy and unreliable pidfile concept"
from the OP, or "pidfile hackery" above.

> This is someone coming to us and saying that it might be a good idea
> to clean up the mess by importing a tiny bit of extra code

That's not even close to an accurate description of what this project
would entail. Have you ever used daemontools?

> I'm not convinced it is a non-starter. I'd fully support Jos if he
> wanted to commit the code and had done the leg work to do it.

One would hope that it would take more than just your support to
entirely change the way that we start and manage services in FreeBSD.

Also, see my followup to Jos' subsequent post.


Doug

--

It's always a long day; 86400 doesn't fit into a short.

Breadth of IT experience, and depth of knowledge in the DNS.
Yours for the right price. :) http://SupersetSolutions.com/

_______________________________________________

Doug Barton

unread,
Jan 17, 2012, 2:50:37 AM1/17/12
to Jos Backus, freebs...@freebsd.org
On 01/16/2012 22:32, Jos Backus wrote:

> I want/need a solution that works in (nearly) all cases and is devoid of
> complex code trying to track state that is already represented elsewhere
> in the system (the process table and the parent/child process
> relationship). I want a solution that can reliably handle a crashing
> server that doesn't clean up its pidfile (the finish script
> functionality in daemontools-encore provides this),

We get it, you want daemontools. It's in the ports, you can have it.

> and I want a unified
> control interface for the services running on a box,

rc.d provides that, and service(8) makes that easier.

> a la launchd or what have you.

We've looked at importing launchd, or something like it. It's not a bad
idea, it's just way more complex than it sounds. And a lot more work
than "hey, let's import daemontools."

If we were going to do something like this I think we should properly
spec out what the goals should be, what the available solutions are, and
what we want our ultimate solution to look like when we're done.

> This isn't about religion but about missing base system
> functionality - the ability to reliably control services running on a box.

And my argument is that we already have that in the base, it's just not
the one you want; and since it's not the one you want you're redefining
"reliably" to suit your needs.

> I thought the motto was "tools, not policies" ;)

Right now you have options (or tools if you will). If the base were
redesigned to use daemontools it would be very difficult to retain those
options.

> And lest people think that I'm just hating on daemontools, I'm not. I
> use it for some things. But converting everything in the base to use it
> is a non-starter, even if we wanted to import it, which I don't see any
> need to do.
>
>
> Straw man. I'm asking for FreeBSD to *support* this functionality out of
> the box, just like OS X, Solaris, AIX and some Linux versions (with
> systemd).

If you can come up with patches to make both options possible out of the
box, I'm sure that people would be interested in reviewing them.

--

It's always a long day; 86400 doesn't fit into a short.

Breadth of IT experience, and depth of knowledge in the DNS.
Yours for the right price. :) http://SupersetSolutions.com/

_______________________________________________

Dag-Erling Smørgrav

unread,
Jan 17, 2012, 3:50:04 AM1/17/12
to Jos Backus, Doug Barton, freebs...@freebsd.org
Jos Backus <j...@catnook.com> writes:
> If FreeBSD had a solid solution out of the box, all this pidfile hackery in
> the base system wouldn't be necessary. I always thought FreeBSD was about
> good engineering. Perpetuating the pidfile mess in the base is not a sign
> of good engineering.

Software written by DJB hardly qualifies as "good engineering". PID
files are well-known and well-tested, we have a solid implementation
with a simple API (pidfile(3)), and a lot of our infrastructure depends
on them (/etc/rc, newsyslog etc.)

DES
--
Dag-Erling Smørgrav - d...@des.no

Dag-Erling Smørgrav

unread,
Jan 17, 2012, 4:00:26 AM1/17/12
to Warner Losh, Doug Barton, freebs...@freebsd.org
Warner Losh <i...@bsdimp.com> writes:
> I'm not convinced it is a non-starter. I'd fully support Jos if he
> wanted to commit the code and had done the leg work to do it. I
> wouldn't support just importing the daemontools and leaving it at
> that. If that's the plan, then leaving it in ports is the best bet.

If we ever want to move in that direction, we should look at launchd,
not daemontools.

DES
--
Dag-Erling Smørgrav - d...@des.no
_______________________________________________

Dag-Erling Smørgrav

unread,
Jan 17, 2012, 4:25:36 AM1/17/12
to Jos Backus, Doug Barton, freebs...@freebsd.org
Jos Backus <j...@catnook.com> writes:
> I want/need a solution that works in (nearly) all cases and is devoid of
> complex code trying to track state that is already represented elsewhere in
> the system (the process table and the parent/child process
> relationship).

Please show me the complex code required to handle pidfiles.

> I want a solution that can reliably handle a crashing server that
> doesn't clean up its pidfile

That's a strawman. Whatever tool you use needs to be able to handle
stale pidfiles anyway.

> I want a unified control interface for the services running on a box,
> a la launchd or what have you.

So extend service(8) to support enabling / disabling services through
/etc/rc.conf.d/<servicename>. Probably no more than an afternoon's
work.

> This isn't about religion but about missing base system functionality
> - the ability to reliably control services running on a box.

The onus is on you to show that we don't already have that.

DES
--
Dag-Erling Smørgrav - d...@des.no

Bruce Evans

unread,
Jan 17, 2012, 7:53:32 AM1/17/12
to Doug Barton, freebs...@freebsd.org
On Mon, 16 Jan 2012, Doug Barton wrote:

> On 01/16/2012 22:32, Jos Backus wrote:

> ...


>> I thought the motto was "tools, not policies" ;)
>
> Right now you have options (or tools if you will). If the base were
> redesigned to use daemontools it would be very difficult to retain those
> options.

>> ...


>> Straw man. I'm asking for FreeBSD to *support* this functionality out of
>> the box, just like OS X, Solaris, AIX and some Linux versions (with
>> systemd).
>
> If you can come up with patches to make both options possible out of the
> box, I'm sure that people would be interested in reviewing them.

Nah, he must also maintain both versions forever :-).

Bruce

Tom Rhodes

unread,
Jan 17, 2012, 9:24:22 AM1/17/12
to Doug Barton, freebs...@freebsd.org

Just use/improve my fscd. I meant to import it but have
just ended up getting too busy. Now, I'm way too busy and
would be more than happy to help anyone bring it in.

That said, happy new year. :P

--
Tom Rhodes

Warner Losh

unread,
Jan 17, 2012, 12:29:53 PM1/17/12
to Doug Barton, freebs...@freebsd.org

On Jan 17, 2012, at 12:34 AM, Doug Barton wrote:

> On 01/16/2012 23:10, Warner Losh wrote:
>>
>> On Jan 16, 2012, at 10:10 PM, Doug Barton wrote:
>>
>>> On 01/16/2012 19:41, Jos Backus wrote:
>>>> On Jan 16, 2012 6:53 PM, "Doug Barton" <do...@freebsd.org>
>>>> wrote:
>>>>>
>>>>> On 01/16/2012 12:53, Jos Backus wrote:
>>>>>>
>>>>>> Thoughts?
>>>>>
>>>>> This is already available in ports.
>>>>
>>>> I realize that.
>>>
>>> Good, then we're done. :)
>>
>> Not necessarily...
>>
>>>> If FreeBSD had a solid solution out of the box, all this pidfile
>>>> hackery in the base system wouldn't be necessary.
>>>
>>> We don't do religious wars here. We especially don't do trollbait
>>> from djb acolytes. The "pidfile hackery" that we currently have
>>> works just fine in the vast majority of cases. The fact that it
>>> doesn't meet some people's ideas of architectural purity is totally
>>> beside the point.
>>
>> This isn't a religious war.
>
> You obviously haven't spent a lot of time dealing with djb'ites. Your
> warning sign should have been "messy and unreliable pidfile concept"
> from the OP, or "pidfile hackery" above.

I have spent time with djb-ites in other areas. I tend to ignore their ranker and focus on the technical issues. I've had issues with pidfiles and such in the past. There are a lot of hacks to get around those issues, and things mostly work. If there's a good alternative that can be demonstrated to work and gain us additional functionality, I'm all for it. I've fought with init() to make it keep important daemons around should they die. I've worked with other systems that make it easy to do and miss that on FreeBSD. It is possible, but not easy. If daemontools makes it easy, we should evaluate it.

>> This is someone coming to us and saying that it might be a good idea
>> to clean up the mess by importing a tiny bit of extra code
>
> That's not even close to an accurate description of what this project
> would entail. Have you ever used daemontools?

I haven't. However, without a fully formed set of patches to test and evaluate technically, it is hard to know if this is a good idea or a bad idea.

>> I'm not convinced it is a non-starter. I'd fully support Jos if he
>> wanted to commit the code and had done the leg work to do it.
>
> One would hope that it would take more than just your support to
> entirely change the way that we start and manage services in FreeBSD.
>
> Also, see my followup to Jos' subsequent post.

And one would hope your pig-headdednes also doesn't keep it out of FreeBSD. Neither you nor I are the final arbiter of what's good for FreeBSD.

Warner

Diane Bruce

unread,
Jan 17, 2012, 1:00:44 PM1/17/12
to Warner Losh, Doug Barton, freebs...@freebsd.org
On Tue, Jan 17, 2012 at 10:29:53AM -0700, Warner Losh wrote:
>
...
>
> I have spent time with djb-ites in other areas. I tend to ignore their rancor and focus on the technical issues. I've had issues with pidfiles and such in the past. There are a lot of hacks to get around those issues, and things mostly work. If there's a good alternative that can be demonstrated to work and gain us additional functionality, I'm all for it. I've fought with init() to make it keep important daemons around should they die. I've worked with other systems that make it easy to do and miss that on FreeBSD. It is possible, but not easy. If daemontools makes it easy, we should evaluate it.

It would be more useful to see what is offered with other systems and see
if we can provide the same or similar functionality with some sort of
'standard' (I know impossible). How much work would it be to simply
add a few scripts as Doug suggests? etc. etc. Instead of just saying
"lets import everything in daemontools", what really is proposed?
How would it impact an embedded system? Is it yet more bloat that should
be delegated to the same pile of important ports as sendmail, bind etc. that
some of us have been trying to get out of base?

- Diane
--
- d...@FreeBSD.org d...@db.net http://www.db.net/~db
Why leave money to our children if we don't leave them the Earth?

Julian Elischer

unread,
Jan 17, 2012, 1:17:05 PM1/17/12
to Warner Losh, Doug Barton, freebs...@freebsd.org
On 1/17/12 9:29 AM, Warner Losh wrote:
> I have spent time with djb-ites in other areas. I tend to ignore
> their ranker and focus on the technical issues. I've had issues with
> pidfiles and such in the past. There are a lot of hacks to get around
> those issues, and things mostly work. If there's a good alternative
> that can be demonstrated to work and gain us additional
> functionality, I'm all for it. I've fought with init() to make it
> keep important daemons around should they die. I've worked with other
> systems that make it easy to do and miss that on FreeBSD. It is
> possible, but not easy. If daemontools makes it easy, we should
> evaluate it.


don't forget other alternatives..

for example we have launchd from apple which is quite a well
tested entry in the "init" space of solutions.

Peter Wemm

unread,
Jan 17, 2012, 1:51:09 PM1/17/12
to Julian Elischer, freebs...@freebsd.org
On Tue, Jan 17, 2012 at 10:17 AM, Julian Elischer <jul...@freebsd.org> wrote:
> On 1/17/12 9:29 AM, Warner Losh wrote:
>>
>>  I have spent time with djb-ites in other areas. I tend to ignore
>>  their ranker and focus on the technical issues. I've had issues with
>>  pidfiles and such in the past. There are a lot of hacks to get around
>>  those issues, and things mostly work. If there's a good alternative
>>  that can be demonstrated to work and gain us additional
>>  functionality, I'm all for it. I've fought with init() to make it
>>  keep important daemons around should they die. I've worked with other
>>  systems that make it easy to do and miss that on FreeBSD. It is
>>  possible, but not easy. If daemontools makes it easy, we should
>>  evaluate it.
>
>
>
> don't forget other alternatives..
>
> for example we have launchd from apple which is quite a well
> tested entry in the "init" space of solutions.

At the risk of prolonging the discussion..

We use daemontools at work and I find it horrid to work with. Really horrid.

However, I do miss a real, pluggable services manager/starter/etc.
launchd springs to mind, if only it wasn't .plist based.

Linux seems to be standardizing on systemd (think of launchd except
made to smell like linux software) as a replacement for everything
including init (just like apple replaced init with launchd).

There has got to be something better and less obnoxious than
daemontools. We don't need another file system abuser like that one.

--
Peter Wemm - pe...@wemm.org; pe...@FreeBSD.org; pe...@yahoo-inc.com; KI6FJV
"All of this is for nothing if we don't go to the stars" - JMS/B5
"If Java had true garbage collection, most programs would delete
themselves upon execution." -- Robert Sewell

Jos Backus

unread,
Jan 17, 2012, 2:37:09 PM1/17/12
to Peter Wemm, freebs...@freebsd.org
Hi Peter,

Thanks for chiming in.

I guess this is a matter of taste. I have automated deployments using
daemontools and found it to integrate quite well with tools like Puppet.

Apache and rsync can now run under it because I argued for it with patches.

But I agree with you otherwise. It doesn't have to be daemontools as long
as there's something which covers most of its functionality, including the
run/finish script stuff, and not having to deal with pidfiles.

Jos


> --
> Peter Wemm - pe...@wemm.org; pe...@FreeBSD.org; pe...@yahoo-inc.com;
> KI6FJV
> "All of this is for nothing if we don't go to the stars" - JMS/B5
> "If Java had true garbage collection, most programs would delete
> themselves upon execution." -- Robert Sewell
> _______________________________________________
> freebs...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-arch
> To unsubscribe, send any mail to "freebsd-arch...@freebsd.org"
>

--

Jos Backus
jos at catnook.com

Baptiste Daroussin

unread,
Jan 17, 2012, 3:18:07 PM1/17/12
to Peter Wemm, Julian Elischer, freebs...@freebsd.org
On Tue, Jan 17, 2012 at 10:51:09AM -0800, Peter Wemm wrote:
> On Tue, Jan 17, 2012 at 10:17 AM, Julian Elischer <jul...@freebsd.org> wrote:
> > On 1/17/12 9:29 AM, Warner Losh wrote:
> >>
> >>  I have spent time with djb-ites in other areas. I tend to ignore
> >>  their ranker and focus on the technical issues. I've had issues with
> >>  pidfiles and such in the past. There are a lot of hacks to get around
> >>  those issues, and things mostly work. If there's a good alternative
> >>  that can be demonstrated to work and gain us additional
> >>  functionality, I'm all for it. I've fought with init() to make it
> >>  keep important daemons around should they die. I've worked with other
> >>  systems that make it easy to do and miss that on FreeBSD. It is
> >>  possible, but not easy. If daemontools makes it easy, we should
> >>  evaluate it.
> >
> >
> >
> > don't forget other alternatives..
> >
> > for example we have launchd from apple which is quite a well
> > tested entry in the "init" space of solutions.
>
> At the risk of prolonging the discussion..
>
> We use daemontools at work and I find it horrid to work with. Really horrid.
>
> However, I do miss a real, pluggable services manager/starter/etc.
> launchd springs to mind, if only it wasn't .plist based.
>

Shouldn't be hard to plug a better format to launchd, something like yaml (Yes I
do like yaml :)) for example.

what was the final statement of the launchd port summer of code?

Can we have something as simple as the current rc with launchd? (for end users)

regards,
Bapt

Xin LI

unread,
Jan 17, 2012, 3:19:32 PM1/17/12
to Baptiste Daroussin, freebs...@freebsd.org

This has to be redone since launchd was later rewritten...

Cheers,
--
Xin LI <del...@delphij.net> https://www.delphij.net/
FreeBSD - The Power to Serve! Live free or die

Jos Backus

unread,
Jan 17, 2012, 4:28:18 PM1/17/12
to Baptiste Daroussin, freebs...@freebsd.org
On Tue, Jan 17, 2012 at 12:18 PM, Baptiste Daroussin <ba...@freebsd.org>wrote:

> Shouldn't be hard to plug a better format to launchd, something like yaml
> (Yes I
> do like yaml :)) for example.
>

I like YAML, too :)


>
> what was the final statement of the launchd port summer of code?
>
> Can we have something as simple as the current rc with launchd? (for end
> users)
>

That would be cool...

Jos


>
> regards,
> Bapt
>

--
Jos Backus
jos at catnook.com

Jos Backus

unread,
Jan 17, 2012, 4:29:52 PM1/17/12
to Xin LI, Baptiste Daroussin, freebs...@freebsd.org
On Tue, Jan 17, 2012 at 12:19 PM, Xin LI <del...@gmail.com> wrote:

> This has to be redone since launchd was later rewritten...
>

How much work would this be, you think? Doesn't launchd rely on some kernel
functionality that FreeBSD doesn't have? Maybe I misremember.

Jos

--
Jos Backus
jos at catnook.com

Jos Backus

unread,
Jan 17, 2012, 4:33:50 PM1/17/12
to Warner Losh, Doug Barton, freebs...@freebsd.org

Thanks, Warner.

I'm perfectly willing to make an effort moving FreeBSD forward in this
area once we can achieve consensus on what moving forward means. I don't
care about the implementation so much as having the functionality available
out of the box. Porting launchd sounds like a good plan.

Jos


> Warner


--
Jos Backus
jos at catnook.com
_______________________________________________

Jos Backus

unread,
Jan 17, 2012, 4:47:21 PM1/17/12
to Dag-Erling Smørgrav, Doug Barton, freebs...@freebsd.org
2012/1/17 Dag-Erling Smørgrav <d...@des.no>

> Jos Backus <j...@catnook.com> writes:
> > If FreeBSD had a solid solution out of the box, all this pidfile hackery
> in
> > the base system wouldn't be necessary. I always thought FreeBSD was about
> > good engineering. Perpetuating the pidfile mess in the base is not a sign
> > of good engineering.
>
> Software written by DJB hardly qualifies as "good engineering". PID
> files are well-known and well-tested, we have a solid implementation
> with a simple API (pidfile(3)), and a lot of our infrastructure depends
> on them (/etc/rc, newsyslog etc.)
>

Just because lots of people have been doing something for a long time that
is widely supported, doesn't mean it's the right thing to do.

As I said before, pidfiles export partial information that is already
available in the process table, without requiring extra cached file system
copies that need to be created/removed, have their permissions managed
carefully and can get stale. daemontools is one implementation that solves
the same problem without using pidfiles. It also gives you the option of a
well-defined startup environment for each service (not tied to the
environment of the caller) and the ability to add fine-grained control over
a service (by manipulating the permissions on the control socket, so select
non-root users can start root services if desired). It also comes with
built-in logging (multilog). Additionally, using the finish script
functionality, policies around crashes can be instituted (e.g. down the
service and send an alert if it crashes more than N times within M
minutes). These are just some of the features that I have used in the past
to run services on hundreds of machines in multiple data centers.

daemontools is surprisingly flexible, and it doesn't require complex
configuration commands or configuration files - a boon when used in
combination with tools like Puppet. That said, I don't care what we pick as
long as we pick something that can do all the above.

Jos


> DES
> --
> Dag-Erling Smørgrav - d...@des.no
>

--

Jos Backus
jos at catnook.com

Jos Backus

unread,
Jan 17, 2012, 4:50:01 PM1/17/12
to Dag-Erling Smørgrav, Doug Barton, freebs...@freebsd.org
2012/1/17 Dag-Erling Smørgrav <d...@des.no>

> Warner Losh <i...@bsdimp.com> writes:
> > I'm not convinced it is a non-starter. I'd fully support Jos if he
> > wanted to commit the code and had done the leg work to do it. I
> > wouldn't support just importing the daemontools and leaving it at
> > that. If that's the plan, then leaving it in ports is the best bet.
>
> If we ever want to move in that direction, we should look at launchd,
> not daemontools.
>

That's fine.

I merely proposed daemontools because I have lots of experience with it, it
is in the public domain and it seems less intrusive than launchd - it would
just be a handful extra binaries people like me could use if they wanted to.

Jos


>
> DES
> --
> Dag-Erling Smørgrav - d...@des.no
> _______________________________________________
> freebs...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-arch
> To unsubscribe, send any mail to "freebsd-arch...@freebsd.org"
>

--

Jos Backus
jos at catnook.com

Jos Backus

unread,
Jan 17, 2012, 4:54:06 PM1/17/12
to Dag-Erling Smørgrav, Doug Barton, freebs...@freebsd.org
2012/1/17 Dag-Erling Smørgrav <d...@des.no>

> Jos Backus <j...@catnook.com> writes:
> > I want/need a solution that works in (nearly) all cases and is devoid of
> > complex code trying to track state that is already represented elsewhere
> in
> > the system (the process table and the parent/child process
> > relationship).
>
> Please show me the complex code required to handle pidfiles.
>

In my solution, no such code exists, so whatever code is there now (the
pidfile library/API) is more complex by definition :)


>
> > I want a solution that can reliably handle a crashing server that
> > doesn't clean up its pidfile
>
> That's a strawman. Whatever tool you use needs to be able to handle
> stale pidfiles anyway.
>

Um, why? The solution I propose doesn't use pidfiles at all.


>
> > I want a unified control interface for the services running on a box,
> > a la launchd or what have you.
>
> So extend service(8) to support enabling / disabling services through
> /etc/rc.conf.d/<servicename>. Probably no more than an afternoon's
> work.
>
> > This isn't about religion but about missing base system functionality
> > - the ability to reliably control services running on a box.
>
> The onus is on you to show that we don't already have that.
>

As I said before, the only thing we have today that will automatically
restart services is init, and it is not flexible enough. That's where
launchd would come in, but that would be a much more invasive change than
just adding the daemontools bits, which would be optional and could be
integrated gradually.

Jos


> DES
> --
> Dag-Erling Smørgrav - d...@des.no
>

--

Jos Backus
jos at catnook.com

Jos Backus

unread,
Jan 17, 2012, 4:57:54 PM1/17/12
to Tom Rhodes, Doug Barton, freebs...@freebsd.org

I looked at fsc briefly but it seems to use pidfiles as well. I also don't
like the BUGS section of fscd.8. The daemontools approach doesn't have this
issue; if a service under daemontools' control dies for whatever reason,
the OS will notify supervise and the finish script (in daemontools-encore)
can perform any user-directed action as necessary.

Jos

> That said, happy new year. :P
>
> --
> Tom Rhodes
>

--

Jos Backus
jos at catnook.com

Jos Backus

unread,
Jan 17, 2012, 5:04:56 PM1/17/12
to Diane Bruce, Doug Barton, freebs...@freebsd.org
On Tue, Jan 17, 2012 at 10:00 AM, Diane Bruce <d...@db.net> wrote:

> On Tue, Jan 17, 2012 at 10:29:53AM -0700, Warner Losh wrote:
> >
> ...
> >
> > I have spent time with djb-ites in other areas. I tend to ignore their
> rancor and focus on the technical issues. I've had issues with pidfiles
> and such in the past. There are a lot of hacks to get around those issues,
> and things mostly work. If there's a good alternative that can be
> demonstrated to work and gain us additional functionality, I'm all for it.
> I've fought with init() to make it keep important daemons around should
> they die. I've worked with other systems that make it easy to do and miss
> that on FreeBSD. It is possible, but not easy. If daemontools makes it
> easy, we should evaluate it.
>
> It would be more useful to see what is offered with other systems and see
> if we can provide the same or similar functionality with some sort of
> 'standard' (I know impossible). How much work would it be to simply
> add a few scripts as Doug suggests? etc. etc. Instead of just saying
> "lets import everything in daemontools", what really is proposed?
> How would it impact an embedded system? Is it yet more bloat that should
> be delegated to the same pile of important ports as sendmail, bind etc.
> that
> some of us have been trying to get out of base?
>

I'm proposing that we add some sort of service management framework (big
words, I know) to FreeBSD, as it seems to be the missing part between the
OS and the services running on the OS such as Apache. The only solution
that can handle automatic service restarts without some form of polling is
init, but it is not flexible enough. I am proposing to add daemontools
(specifically, daemontools-encore) because it could be added without
affecting anything else in the base so it would be an easy import: don't
use it if you don't want to. daemontools has not undergone many changes in
recent years so if we were to import it it would likely not change much
thereafter, requiring little maintenance.

That said, if we feel that importing launchd is the better way to go, I'm
all for it. I just want us to add something to fill the gap.

Jos

>
> - Diane
> --
> - d...@FreeBSD.org d...@db.net http://www.db.net/~db
> Why leave money to our children if we don't leave them the Earth?
> _______________________________________________
> freebs...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-arch
> To unsubscribe, send any mail to "freebsd-arch...@freebsd.org"
>

--

Jos Backus
jos at catnook.com

Doug Barton

unread,
Jan 17, 2012, 5:07:06 PM1/17/12
to Jos Backus, freebs...@freebsd.org
On 01/17/2012 14:04, Jos Backus wrote:
> I just want us to add something to fill the gap.

I think that you have made your perspective quite clear at this point.

--

It's always a long day; 86400 doesn't fit into a short.

Breadth of IT experience, and depth of knowledge in the DNS.
Yours for the right price. :) http://SupersetSolutions.com/

_______________________________________________

Jos Backus

unread,
Jan 17, 2012, 5:07:31 PM1/17/12
to Ivo Vachkov, Doug Barton, freebs...@freebsd.org
On Tue, Jan 17, 2012 at 1:56 PM, Ivo Vachkov <ivo.v...@gmail.com> wrote:

> Wouldn't it be more logical to first:
> 1) Define what a modern start/boot/service control system should do?
> 2) Define technological and architectural constraints?
> ... and only then jump to "port *this*" kind of discussions ...
>
> I know of at least one successful commercial project to port launchd on
> FreeBSD (6.x and 7.x), but still there are also others: initNG, eINIT,
> Upstart, Service Management Facility, etc.
>
> I don't want to start another flamewar here, i'm aware of license issues,
> dead code, commercial issues and so on, I just want to point out that there
> are other options and IMHO the focus should not be on what to port, but
> what to develop that suits our needs.
>

If people feel that's the right thing to talk about first, by all means,
sure.

Jos


>
>
>
>> > Warner
>>
>>
>>
>>
>> --
>> Jos Backus
>> jos at catnook.com
>> _______________________________________________
>> freebs...@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-arch
>> To unsubscribe, send any mail to "freebsd-arch...@freebsd.org"
>>
>
>
>

> --
> Ivo Vachkov

Jos Backus

unread,
Jan 17, 2012, 5:08:50 PM1/17/12
to Doug Barton, freebs...@freebsd.org
On Tue, Jan 17, 2012 at 2:07 PM, Doug Barton <do...@freebsd.org> wrote:

> On 01/17/2012 14:04, Jos Backus wrote:
> > I just want us to add something to fill the gap.
>
> I think that you have made your perspective quite clear at this point.
>

Excellent! I'd like to think that good communication is half the struggle :)

Jos


--
Jos Backus
jos at catnook.com

Warner Losh

unread,
Jan 17, 2012, 5:09:06 PM1/17/12
to Jos Backus, Doug Barton, freebs...@freebsd.org

On Jan 17, 2012, at 2:33 PM, Jos Backus wrote:
> Let's not dismiss this out of hand.
>
> Thanks, Warner.
>
> I'm perfectly willing to make an effort moving FreeBSD forward in this area once we can achieve consensus on what moving forward means. I don't care about the implementation so much as having the functionality available out of the box. Porting launchd sounds like a good plan.

Sounds like a good idea. I know the current init stuff is weak and has to be worked around. /etc/rc.d has no interaction with init at this level, which is what the problem is. init has the ability to keep things alive, but little flexibility. /etc/rc.d has the ability to launch a bunch of stuff, but little ability to keep things alive. Any improvement in this area is needed. launchd is a better solution still, and I look forward to seeing it in the tree if you get it working.

But to reiterate my point: the judgement will be on the result of the work. If it is done poorly, it may not make it into the tree. If it turns out that launchd has some fundamental incompatibility with FreeBSD, you may have to do a lot of work for no gain in the tree. On the other hand, it should work well and likely is less work than integrating daemontools completely into the system (key point here is completely).

Warner

Ivo Vachkov

unread,
Jan 17, 2012, 4:56:45 PM1/17/12
to Jos Backus, Doug Barton, freebs...@freebsd.org
On Tue, Jan 17, 2012 at 11:33 PM, Jos Backus <j...@catnook.com> wrote:

Wouldn't it be more logical to first:
1) Define what a modern start/boot/service control system should do?
2) Define technological and architectural constraints?
... and only then jump to "port *this*" kind of discussions ...

I know of at least one successful commercial project to port launchd on
FreeBSD (6.x and 7.x), but still there are also others: initNG, eINIT,
Upstart, Service Management Facility, etc.

I don't want to start another flamewar here, i'm aware of license issues,
dead code, commercial issues and so on, I just want to point out that there
are other options and IMHO the focus should not be on what to port, but
what to develop that suits our needs.

> > Warner


>
>
>
>
> --
> Jos Backus
> jos at catnook.com
> _______________________________________________
> freebs...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-arch
> To unsubscribe, send any mail to "freebsd-arch...@freebsd.org"
>

--
Ivo Vachkov

Tom Rhodes

unread,
Jan 17, 2012, 6:52:47 PM1/17/12
to Jos Backus, trh...@freebsd.org, do...@freebsd.org, freebs...@freebsd.org

That bug could be fixed (IIRC what it was) - and it does most of the
legwork you're talking about, removing pidfile requirements would
probably be trivial for someone with a free evening. Regardless, I
tried to find a common ground with launchd and nothing, the
community is more than welcome to make changes. Maybe I'll
just test build and bring it in, then people can make it work the
way they'd like. ;)

--
Tom Rhodes

Jos Backus

unread,
Jan 17, 2012, 8:39:04 PM1/17/12
to Tom Rhodes, do...@freebsd.org, freebs...@freebsd.org
On Tue, Jan 17, 2012 at 3:52 PM, Tom Rhodes <trh...@freebsd.org> wrote:

> [snip]


> That bug could be fixed (IIRC what it was) - and it does most of the
> legwork you're talking about, removing pidfile requirements would
> probably be trivial for someone with a free evening. Regardless, I
> tried to find a common ground with launchd and nothing, the
> community is more than welcome to make changes. Maybe I'll
> just test build and bring it in, then people can make it work the
> way they'd like. ;)
>

I'm not opposed to this if the consensus is that we want to try this.

Here are some of the things I like about daemontools that I would want any
accepted solution to support, as I have actively used all these features in
the past:
- Decouple running of the service from the user controlling the service.
With daemontools, this is handled by the service run script being run by
supervise in response to a command sent across a UNIX domain socket. No
environment variables or other user state can leak into the service process
by default.
- The ability for arbitrary sets of (non-root) users to control a service.
With daemontools this can be achieved by manipulating the UNIX domain
socket permissions for a service.
- The ability to reliably deal with crashing processes and be able to take
action upon signals/non-zero exits. With daemontools-encore, the finish
script lets one fine-tune these cases, and standard daemontools can
automatically restart a service.
- Simple command interface to bring a service up or down, up once
(one-shot) or send signals. With daemontools, the svc command is used for
this.
- Simple command interface to see the status of a service, including its
uptime and pending status. E.g. with daemontools, svstat will display
service status and uptime, including if a service wants to go down.
- Simple service configuration. In daemontools, the file system is used.
This makes it possible to use tools such as `echo', `rm' and `ln -s' to
manipulate and configure services independently from each other. I don't
like a single shared configuration file such as /etc/inittab because it
makes it difficult to edit services safely as the wrong edit can affect all
services (think Puppet). E.g. with daemontools, enabling a service can be
as simple as an `ln -s' command. Editing the state of one service should
not be able to affect any other service.
- Handle logging of stdout/stderr. daemontools uses the associated log
service for this, usually with multilog.
- No reliance on pidfiles. In daemontools, supervise is the parent of a
service so it already knows its child's pid.

If fsc can be made to do all these things too and we feel more comfortable
with it than importing daemontools, that's great.

Jos

--
Jos Backus
jos at catnook.com

Ed Maste

unread,
Jan 17, 2012, 8:57:21 PM1/17/12
to Jos Backus, freebs...@freebsd.org
On Tue, Jan 17, 2012 at 01:29:52PM -0800, Jos Backus wrote:

> How much work would this be, you think? Doesn't launchd rely on some kernel
> functionality that FreeBSD doesn't have? Maybe I misremember.

Are you perhaps thinking of systemd, which does use a whole bunch of
Linux-specific features?

There was a GSoC project to port launchd in the past; I have no idea
how much it has changed since then.

http://wiki.freebsd.org/launchd

-Ed

Jos Backus

unread,
Jan 17, 2012, 9:17:40 PM1/17/12
to Ed Maste, freebs...@freebsd.org
On Tue, Jan 17, 2012 at 5:57 PM, Ed Maste <ema...@freebsd.org> wrote:

> On Tue, Jan 17, 2012 at 01:29:52PM -0800, Jos Backus wrote:
>
> > How much work would this be, you think? Doesn't launchd rely on some
> kernel
> > functionality that FreeBSD doesn't have? Maybe I misremember.
>
> Are you perhaps thinking of systemd, which does use a whole bunch of
> Linux-specific features?
>

Probably :) I guess part of the launchd porting effort was removing stuff
that FreeBSD doesn't support out of the box, such as zeroconf/Bonjour.


> There was a GSoC project to port launchd in the past; I have no idea
> how much it has changed since then.
>
> http://wiki.freebsd.org/launchd


I found the wiki and the Perforce soc2005 branch on perforce.freebsd.org,
as well as the official launchd Subversion repo at
http://launchd.macosforge.org. There's also
https://github.com/rtyler/launchd-for-freebsd which seems to based on the
Subversion repo but at an older revision. Only the Perforce branch seems to
have any FreeBSD build glue. I guess a plan would be to port the changes
made to Subversion trunk since the Perforce snapshot was taken to the
Perforce branch. It looks like launchd hasn't changed in several years.

After looking at the launchd code for a little bit, it looks much more
complicated than daemontools, which in turn worries me in terms of its
expected robustness. One of the nice properties of daemontools is that the
individual pieces are fairly small and have been proven in the field to
work well.

Specifically, I'm worried that it will be harder to create a quality port
of launchd compared to daemontools, and as Warner has hinted at, the port
needs to work well for it to be accepted. It seems to me that a working
port of daemontools is easier to achieve because the code is (much)
simpler. Hence the chances of a solution to be accepted seem higher when
using daemontools as a base. But maybe I'm overestimating the amount of
work needed to get launchd into good enough shape to be committed.

Jos


>
> -Ed
>

--
Jos Backus
jos at catnook.com

David Xu

unread,
Jan 19, 2012, 8:50:51 AM1/19/12
to Jos Backus, Dag-Erling Smørgrav, Doug Barton, freebs...@freebsd.org
On 2012/1/18 5:47, Jos Backus wrote:
> 2012/1/17 Dag-Erling Sm�rgrav<d...@des.no>
This sounds like a very cool tool, looks like these ideas are widely
adopted,
I found some of the concepts are also in erlang OTP. ;-)

Regards,
David Xu

Reply all
Reply to author
Forward
0 new messages