sysinst split project - The Configuration File
Silas Silva
(I'm Cc'ing this to current-users@, although I don't know if this is
right - if not, tell me so I can e-mail always on tech-install only).
There is a long time I posted my e-mail in tech-install@ about my wish
of beginning the sysinst split project. I believed I could provide an
initial agenda for the project, but I barely could start it (no time!).
My friend (I'll let introduce himself, if he wishes) and I started with
a basic configuration file that we want to discuss with you and, when
there will be some consensus around it, we'll close its specification in
a BNF grammar and we'll start to implement the parser lib in Lex, Yacc
and C.
I posted it here: http://pastebin.com/f5ec7a9f
We've taken a look at the 2008 gsoc project Mark pointed us. There is
interesting things to get from there, but we found its syntax is a bit
confused, so we made the one I linked above. We tried to keep "fields"
there (size, start, FStype) like they show up in classical sysinst.
Please, take a look and make suggestions.
When we get a lib ready for use, we'll start the actual sysinst split.
We don't know yet what will be done, possibly one of the two
possibilities:
- Import Dragonfly BSD installer (which looks very interesting, but IMO,
has some problems to combine with the philosophy we are adopting for
the sysinst split project (the existence of a configuration file)).
- Get classical sysinst and split it in backend and frontend, like the
original proposal and according to the abandoned 2008 gsoc project.
Please, review the sysinst.conf file proposal and tell me what we can do
to make it better.
Thank you very much!
--
Silas Silva
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-...@muc.de
Aaron J. Grier
> I posted it here: http://pastebin.com/f5ec7a9f
how will partitions vs disklabel be handled for architectures which
don't differentiate between the two? sparc, alpha, decstation don't
care about i386-style partitions.
add provisions for rc.conf settings and extra files in /etc?
I think this is a very worthwhile project and will make it easier to
deploy large amounts of NetBSD hosts.
--
Aaron J. Grier | "Not your ordinary poofy goof." | agr...@poofygoof.com
Silas Silva
> On Tue, Nov 24, 2009 at 12:18:23PM -0200, Silas Silva wrote:
> don't differentiate between the two? sparc, alpha, decstation don't
> care about i386-style partitions.
Hey Aaron, thank you very much for your reply.
We actually know only i386 and xen ports and we have no experience with
other architectures (I just installed NetBSD on an alpha once, but
nothing more). We intend to make an alpha version working only for i386
first, but of course the new sysinst will need to be deployed on all
architectures when we have a stable version ready to be commited to CVS.
Well, I don't think any sysinst.conf file for specific architectures
will differ too much. We could just took out (or skip) the "partitions"
section on configuration files for these architectures (alpha, sparc,
etc.). Just an idea.
> add provisions for rc.conf settings and extra files in /etc?
Yeah, that would be great. What's better?:
1) Make /etc files contents available in sysinst.conf, e. g.:
# Something like that:
settings {
file {
path: /etc/rc.conf
content: {
multi-line content?
}
}
}
2) Or just link them so the backend copies from the sets, CD or whatever,
to the host. E. g.:
# Something like that:
settings {
file {
source: /mnt/cdrom/defaults/etc/rc.conf
destination: /etc/rc.conf
}
file {
source: http://localserver/defaults/etc/sysctl.conf
destination: /etc/sysctl.conf
}
}
Both options seem fine for me. Maybe it is a good idea to support both.
What do you think?
matthew sporleder
Couldn't you save yourself some trouble by using a property list (
proplib(3) ) as the format?
Silas Silva
> On Wed, Nov 25, 2009 at 08:44:06AM -0500, matthew sporleder wrote:
> > Couldn't you save yourself some trouble by using a property list (
> > proplib(3) ) as the format?
>
Thank you for the tip Matthew and Martin. I've heard about proplib, but
never used that. I quickly searched for it on the internet and found
that NetBSD has a very good implementation of it.
It seems it implements a lot of data structures that we would have to
implement "from the ground" (I hope this English idiom is right) and it
is used on some utilities, like ifconfig, as I found in
opengrok.netbsd.org.
Yeah, that is a very good tip, thank you very much. Of course we can't
use its import/export to/from files feature, since, if I saw it
correctly, it works with XML, which is not "editor-friendly" :-)
Thank you.
--
Silas Silva
David Young
>
> (I'm Cc'ing this to current-users@, although I don't know if this is
> right - if not, tell me so I can e-mail always on tech-install only).
>
> There is a long time I posted my e-mail in tech-install@ about my wish
> of beginning the sysinst split project. I believed I could provide an
> initial agenda for the project, but I barely could start it (no time!).
>
> My friend (I'll let introduce himself, if he wishes) and I started with
> a basic configuration file that we want to discuss with you and, when
> there will be some consensus around it, we'll close its specification in
> a BNF grammar and we'll start to implement the parser lib in Lex, Yacc
> and C.
>
> I posted it here: http://pastebin.com/f5ec7a9f
I have a couple of urgent suggestions:
Coordinate with Lloyd Parkes and Masao Uebayashi. IMO, system
installation is essentially the same as building system images (see
LLoyd's Miniaturize NetBSD GSoC 2009 project), but installation is
"on-line" while image-building is "off-line" (for lack of better
terms). They both need a configuration file format. They both need
to layout disks, add users/groups, add 3rd-party software, and install
configuration files in /etc/.
Use XML for the configuration. That ensures that the configuration is
queryable/transformable with NetBSD's forthcoming xmltools. I know that
writing XML by hand is objectionable, so establish a one-to-one mapping
from your user-friendly format to an XML format. Provide a tool to/from
the XML format.
> - Get classical sysinst and split it in backend and frontend, like the
> original proposal and according to the abandoned 2008 gsoc project.
Yes, please.
Dave
--
David Young OJC Technologies
dyo...@ojctech.com Urbana, IL * (217) 278-3933
Manuel Bouyer
> Alexander Nasonov writes:
> > David Young wrote:
> > > Use XML for the configuration. That ensures that the configuration is
> > > queryable/transformable with NetBSD's forthcoming xmltools.
> >
> > instead.
>
> How many people know XML? How many people know lua?
>
> How many tools can process XML? How many tools can "process" lua?
Can you write programs in XML?
--
Manuel Bouyer <bou...@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
matthew sporleder
> On Sun, Dec 13, 2009 at 03:56:44PM +0100, Volker A. Brandt wrote:
>> Alexander Nasonov writes:
>> > David Young wrote:
>> > > queryable/transformable with NetBSD's forthcoming xmltools.
>> >
>> > instead.
>>
>> How many people know XML? How many people know lua?
>>
>> How many tools can process XML? How many tools can "process" lua?
>
> Can you write programs in XML?
>
Of course you can, but why would you want to when every programming
language is so good at processing them already?
http://en.wikipedia.org/wiki/Category:XML-based_programming_languages
See also:
http://en.wikipedia.org/wiki/OpenLazlo
http://en.wikipedia.org/wiki/XSLT
http://en.wikipedia.org/wiki/XUL
http://en.wikipedia.org/wiki/MXML
http://en.wikipedia.org/wiki/Extensible_Application_Markup_Language
http://en.wikipedia.org/wiki/EMML
Alexander Nasonov
> How many tools can process XML? How many tools can "process" lua?
We are here to change the world!
Alex
Greg A. Woods
Subject: Re: sysinst split project - The Configuration File
>
> Alexander Nasonov writes:
> > David Young wrote:
> > > queryable/transformable with NetBSD's forthcoming xmltools.
> >
> > instead.
>
> How many people know XML? How many people know lua?
I'm not sure that's relevant, but regardless that's an apples
vs. oranges question.
You must ask how many people know XML _plus_ the relevant DTDs.
LUA is one language -- XML is a family of languages.
As for trying to count the number of people who "know" one language or
another, well that's a specious argument in this context. XML syntax
might be easy to know, and a given XML DTD might be simple to
understand, but SGML style syntax overall is a _horrible_ syntax for
both humans and computers to use. The very fact that you need to use a
set of tools to manipulate it in any way, and the fact these tools
consist of relatively massive amounts of code, is sure evidence of this
fact.
> How many tools can process XML? How many tools can "process" lua?
That question is out of scope for this problem.
You only need a select set of tools for this given task. Indeed one
needs far fewer tools in this context to make use of lua than one would
need to make use of a relevant XML DTD. I.e. with lua one needs only
the built-in embedded interpreter (and a text editor) to use lua. It's
already suggested that one would need at least xmltools, an XML-capable
editor, plus also perhaps some other tools to make use of an XML DTD.
However I dare say a lot more human programmers will have an easier time
of "processing" lua than they will of processing a private new XML DTD.
I think the point also is that sysinst(8) _really_ NEEDS an embedded
scripting language anyway, so also adding a new private XML DTD to the
mess would be a very unnecessary thing to do. Even if one could find a
widely used DTD to use which matches the requirements needed here would
still mean adding a whole lot of extra crud that's going to be totally
unnecessary when a proper embedded scripting language is also added.
Why does everyone seem want to shout "XML" in answer to such a wide
variety of questions when that isn't even a proper or complete answer to
any possible question!!?!?!? Only an answer along the lines of
"such-and-such XML DTD" would be even close to suitable and complete.
Meanwhile throwing XML at almost any problem means throwing a huge
amount of extra code and complex tools at a problem where a simple
human-friendly embedded scripting language would easily provide more
than enough features and support to solve the problem, _especially_ in
this case. If XML in any form is the best answer you can come up with
then you've probably asked the wrong question in the first place --
either that or it's the only tool in your toolbox.
--
Greg A. Woods
Planix, Inc.
Volker A. Brandt
Thanks for taking the time for an in-depth answer.
> > How many people know XML? How many people know lua?
>
> I'm not sure that's relevant, but regardless that's an apples
> vs. oranges question.
Of course. :-)
> You must ask how many people know XML _plus_ the relevant DTDs.
>
> LUA is one language -- XML is a family of languages.
Hmmmm... yes and no. XML is a method to structure data. Together
with XSLT and other things, you can implement languages.
> As for trying to count the number of people who "know" one language or
> another, well that's a specious argument in this context. XML syntax
> might be easy to know, and a given XML DTD might be simple to
> understand, but SGML style syntax overall is a _horrible_ syntax for
> both humans and computers to use.
I respectfully disagree. I find it not so bad, and computers don't care.
> The very fact that you need to use a
> set of tools to manipulate it in any way, and the fact these tools
> consist of relatively massive amounts of code, is sure evidence of this
> fact.
My tool is called emacs. :-) And you would need an interpreter for any
scripting language, i.e. a "tool to manipulate" the script.
> You only need a select set of tools for this given task. Indeed one
> needs far fewer tools in this context to make use of lua than one would
> need to make use of a relevant XML DTD. I.e. with lua one needs only
> the built-in embedded interpreter (and a text editor) to use lua. It's
> already suggested that one would need at least xmltools, an XML-capable
> editor, plus also perhaps some other tools to make use of an XML DTD.
You would need an editor for Lua. OTOH you can write XML in vi.
> However I dare say a lot more human programmers will have an easier time
> of "processing" lua than they will of processing a private new XML DTD.
We are talking about a configuration file. It will be touched by humans,
and processed by machine. This holds true for Lua and XML both. It is
just text that to you need to read and understand.
> I think the point also is that sysinst(8) _really_ NEEDS an embedded
> scripting language anyway, so also adding a new private XML DTD to the
> mess would be a very unnecessary thing to do. Even if one could find a
> widely used DTD to use which matches the requirements needed here would
> still mean adding a whole lot of extra crud that's going to be totally
> unnecessary when a proper embedded scripting language is also added.
So does that mean that Lua could not process XML?
> Why does everyone seem want to shout "XML" in answer to such a wide
> variety of questions when that isn't even a proper or complete answer to
> any possible question!!?!?!?
Maybe because "everyone" is not interested in a "pure" way for its own
sake, but cares more for established standards, easy interoperability,
and accessibility for the non-guru level user.
> Only an answer along the lines of
> "such-and-such XML DTD" would be even close to suitable and complete.
Of course a complete answer would be to provide a DTD or schema and a
parser implementation. In that sense, no answer could be complete unless
it provided the finished sysinst rewrite. We are not talking on that
level.
> Meanwhile throwing XML at almost any problem means throwing a huge
> amount of extra code and complex tools at a problem where a simple
> human-friendly embedded scripting language would easily provide more
> than enough features and support to solve the problem, _especially_ in
> this case.
It is quite possible that LUA would solve this isolated problem more elegantly.
> If XML in any form is the best answer you can come up with
> then you've probably asked the wrong question in the first place --
> either that or it's the only tool in your toolbox.
I find this remark uncalled for. One of the things that steered me
away from working on NetBSD is that unhealthy tendency in this community
to propose one single "true" way and dismiss everything else, coupled
with a severe case of NIH syndrome. (Not to dramatize things -- of course
the main reason was lack of time, as usual. Code speaks louder than
words.)
Anyway, I would have liked to see a "parameterized sysinst + native
packaging tool + system packages + automated netboot" installation system
for NetBSD. It would have gone a long way towards acceptance of NetBSD
outside the "hardcore" user community.
Regards -- Volker
--
------------------------------------------------------------------------
Volker A. Brandt Consulting and Support for Sun Solaris
Brandt & Brandt Computer GmbH WWW: http://www.bb-c.de/
Am Wiesenpfad 6, 53340 Meckenheim Email: v...@bb-c.de
Handelsregister: Amtsgericht Bonn, HRB 10513 Schuhgröße: 45
Geschäftsführer: Rainer J. H. Brandt und Volker A. Brandt
Greg A. Woods
Subject: Re: sysinst split project - The Configuration File
>
>
> with XSLT and other things, you can implement languages.
Well, to me language doesn't necessarily imply that it represents
instructions which can be executed. A "language" can simply describe
something as well. If I understand correctly you're using "languages"
in this context to describe something that can be used to write
executable instructions for computers.
So, well, XML is, as I understand it, mostly just a semantic-free syntax
and a framework for defining the semantics that may be added to it or
used with it. The DTD brings structure and semantics to whatever is
represented in the XML syntax.
I guess "XSLT and other things" would additionally implement what you're
calling languages here, i.e. something that can cause actions to be
carried out based on the semantics of the language.
> > As for trying to count the number of people who "know" one language or
> > another, well that's a specious argument in this context. XML syntax
> > might be easy to know, and a given XML DTD might be simple to
> > understand, but SGML style syntax overall is a _horrible_ syntax for
> > both humans and computers to use.
>
> I respectfully disagree. I find it not so bad, and computers don't care.
Indeed there are far too many people who have become accustomed to the
SGML-style of syntax. We can thank the use of HTML for the WWW for
that. It really is unfortunate that the chosen foundation for
hypertext markup was SGML. I think that's a very sad accident of
history. That's my personal opinion, of course, but seems I'm not alone
in thinking such things, and it is based on my own extensive experience
in this field.
However even if as a result of this accident many people, including
yourself, are familiar with SGML-like syntax and you don't really find
it all that bad, that still doesn't make it any less difficult overall
for either humans or computers to use. :-)
As for the computers not caring, well perhaps they don't directly care
so much, but someone still has to write and maintain the code that makes
them capable of handling XML and DTDs and such, and it's easy to show
that such code is far larger and far more complex and far more difficult
to write and maintain than many other options, especially other
domain-specific solutions. Humans do often care when their computers
suddenly need more resources to solve a given problem.
> > The very fact that you need to use a
> > set of tools to manipulate it in any way, and the fact these tools
> > consist of relatively massive amounts of code, is sure evidence of this
> > fact.
>
> My tool is called emacs. :-) And you would need an interpreter for any
> scripting language, i.e. a "tool to manipulate" the script.
No doubt emacs can be used to include subsystems which can properly
manipulate XML structured data. However that still means including a
whole lot more "unnecessary" code and tools.
Sure, XML experts can no doubt manually manipulate XML with any plain
text editor, even ed(1). I can write and manipulate machine code as raw
numbers too. That doesn't mean it's easy, or that I should.
The point of a good computer language, be it one intended for writing
machine instructions, or be it one for writing structured data, is that
it makes a good human interface to the computer. An ideal computer
language is easy for a human to learn natively and it is also easy for a
computer to parse and make use of. Both aspects are important!
> We are talking about a configuration file. It will be touched by humans,
> and processed by machine. This holds true for Lua and XML both. It is
> just text that to you need to read and understand.
Exactly -- it needs to be expressed in a way that's easy for both humans
and for computers to parse and understand.
I will continue to claim that lua, or awk, or something equally simple
in syntax and semantics is far better for both humans and computers.
> So does that mean that Lua could not process XML?
Why would anyone even dream of layering XML on top of a language that
can already be used very easily to describe and write structured data?
I believe such unnecessary complexity and layering is not good for
either humans or computers!
To me it also begs the question of whether there's some hidden agenda
somehow causing XML to be propped up as an answer. Like trying to get
it's foot in the door, or something like that.
XML might, at times, be a good answer for data interchange between
disparate environments, though even then I think there are much better
solutions (though I agree that ASN.1 is probably not one of them), since
even then one must pre-suppose that one end of the interchange or
another must somehow already be tied inextricably to using XML -- or at
least feel that they are. ;-)
> > If XML in any form is the best answer you can come up with
> > then you've probably asked the wrong question in the first place --
> > either that or it's the only tool in your toolbox.
>
> I find this remark uncalled for. One of the things that steered me
> away from working on NetBSD is that unhealthy tendency in this community
> to propose one single "true" way and dismiss everything else, coupled
> with a severe case of NIH syndrome. (Not to dramatize things -- of course
> the main reason was lack of time, as usual.
Sorry, but I far too often see XML champions jump up and down and posit
XML as the only acceptable solution to a problem without giving any good
explanation as to why it's even in the running, never mind explaining
why they think it's the _only_ acceptable solution. This happens
everywhere, it's not just a phenomenon restricted to the NetBSD world.
> Code speaks louder than
> words.)
Indeed, which is I think why lua was proposed as the best suitable
extension language and configuration language for sysinst, since as I
understand it there's already a prototype implementation using lua.
Note I'm not saying lua is the best or only solution here either. I
just think XML is the wrong solution and that there are better solutions
available, including lua.
> Anyway, I would have liked to see a "parameterized sysinst + native
> packaging tool + system packages + automated netboot" installation system
> for NetBSD. It would have gone a long way towards acceptance of NetBSD
> outside the "hardcore" user community.
Indeed -- we definitely agree on that topic!
Jan Danielsson
[---]
>> Anyway, I would have liked to see a "parameterized sysinst + native
>> packaging tool + system packages + automated netboot" installation system
>> for NetBSD. It would have gone a long way towards acceptance of NetBSD
>> outside the "hardcore" user community.
>
> Indeed -- we definitely agree on that topic!
I've seen a few sysinst discussions in the past, but I haven't seen
anyone bring http://www.bsdinstaller.org/ up. (I'm not saying it's a
viable route for NetBSD, but I'm wondering if the people working on
sysinst have taken a look at it).
To sum up the things which appeal to me:
- Scriptable (in lua, as a matter of fact)
- Separated frontend and backend
- It's been around for a few years, and appears to be stable enough
to be adopted by other projects.
As for problems, I assume it may be too "bloated" for some platforms.
--
Kind regards,
Jan Danielsson
Marc Balmer
Am 14.12.2009 um 21:35 schrieb Jean-Yves Migeon:
> On 12/14/09 18:57, Volker A. Brandt wrote:
> [snip]
>>> As for trying to count the number of people who "know" one
>>> language or
>>> another, well that's a specious argument in this context. XML
>>> syntax
>>> might be easy to know, and a given XML DTD might be simple to
>>> understand, but SGML style syntax overall is a _horrible_ syntax for
>>> both humans and computers to use.
>>
>> I respectfully disagree. I find it not so bad, and computers don't
>> care.
>
> While computers do not care, XML processing is heavy, given all the
> tools required to parse just one simple XML file together with its
> DTD/Schema.
>
> I have to manually tweak XML config files at $DAYJOB on a weekly
> basis, used in an environment that generates a configuration,
> Makefiles and ROM image for embedded devices. It is unbelievably
> heavy, and inconvenient to edit. While I understand that some XML
> files can be structured in a human readable and editable way, the
> vast majority of the files I have seen are just a pain.
>
> Especially when you throw in markup languages like YAML or JSON
> which are, IMO, much easier to read than plain XML.
>
> Anyway, I hope you won't end like HAL and its fdi files. While
> xorg.conf is not something expected to be understandable easily (way
> behind sendmail anyway ;) ), it was easily readable. Now that X.org
> uses HAL, we didn't win on the "understandable" side, but IMHO,
> regress on the "readable" one.
>
>>> I think the point also is that sysinst(8) _really_ NEEDS an embedded
>>> scripting language anyway
>
> +1
>
>>> Why does everyone seem want to shout "XML" in answer to such a wide
>>> variety of questions when that isn't even a proper or complete
>>> answer to
>>> any possible question!!?!?!?
>>
>> Maybe because "everyone" is not interested in a "pure" way for its
>> own
>> sake, but cares more for established standards, easy
>> interoperability,
>> and accessibility for the non-guru level user.
>
> "When all you have is a hammer, everything looks like a nail" syndrom.
>
> What about JSON or YAML then? Do you consider them as "established
> standards, easy interoperability, and accessibility for the non-guru
> level user" ?
>
>> Anyway, I would have liked to see a "parameterized sysinst + native
>> packaging tool + system packages + automated netboot" installation
>> system
>> for NetBSD. It would have gone a long way towards acceptance of
>> NetBSD
>> outside the "hardcore" user community.
>
> I very much agree.
>
Lua would make a lot of sense, since Lua can also be used to drive the
overall logic and user interface. Lua should be embedded in sysinst
to drive it.
Jean-Yves Migeon
> [snip]
> Lua would make a lot of sense, since Lua can also be used to drive the
> overall logic and user interface. Lua should be embedded in sysinst to
> drive it.
I wasn't much in favor of lua first hand, but I agree, it _really_ makes
sense for such use cases.
I did not know about bsdinstaller though, I am wondering how much work
it would need to fit NetBSD's needs, or if it integrates properly with
the sysinst's split project from GSoC.
--
Jean-Yves Migeon
jeanyve...@free.fr
David Young
> On 12/14/09 18:57, Volker A. Brandt wrote:
> [snip]
> >>another, well that's a specious argument in this context. XML syntax
> >>might be easy to know, and a given XML DTD might be simple to
> >>understand, but SGML style syntax overall is a _horrible_ syntax for
> >>both humans and computers to use.
> >
>
> tools required to parse just one simple XML file together with its
> DTD/Schema.
Jean-Yves,
Most of the existing XML tools are too CPU- and memory-intensive, I
agree, but NetBSD happens to be the center for improvements. Producing
a lightweight alternative for XML processing was the concern of a NetBSD
GSoC 2009 project that I mentored. Xmlgrep, for search/extraction
on XML streams, is very usable, and a program called xmlsed, for
transforming XML streams, is under development. By design, xmlgrep
is FAST and it uses memory conservatively. Furthermore, the xmlgrep
binary and its light dependencies are quite a bit smaller than you might
expect:
% ls -l `which xmlgrep`
-rwxr-x--- 1 dyoung wasabi 47K Nov 16 17:57 /home/dyoung/scratch-cuw/xmltools/xmlgrep/xmlgrep
% ldd `which xmlgrep`
/home/dyoung/scratch-cuw/xmltools/xmlgrep/xmlgrep:
-lexpat.1 => /usr/pkg/lib/libexpat.so.1
-lc.12 => /usr/lib/libc.so.12
% ls -l /usr/pkg/lib/libexpat.so.1.5.2
-rwxr-xr-x 1 root wheel 155K Jun 18 2008 /usr/pkg/lib/libexpat.so.1.5.2
Dave
--
David Young OJC Technologies
dyo...@ojctech.com Urbana, IL * (217) 278-3933
--
Thor Lancelot Simon
>
> Jean-Yves,
>
> Most of the existing XML tools are too CPU- and memory-intensive, I
> agree, but NetBSD happens to be the center for improvements.
Nonetheless, this seems like an application where people are likely to
have to edit the configuration files by hand. I would like to register,
again, my strong objection to using XML -- or any other format which has
XML's repugnant feature of token delimiters with semantic meaning, which
is about as bad for human usability as possible -- in any application in
NetBSD in which it may be reasonably expected that humans will have to or
choose to hand-edit the chosen data format.
XML is fine for many purposes. Not, please, for any purposes in which
human beings are likely to end up editing by hand.
Having to recursively do _semantic_ procesing just to figure out where the
boundaries of the data elements are is a syntactical decision only a
standards committee could love.
Thor
David Young
> On Mon, Dec 14, 2009 at 04:32:13PM -0600, David Young wrote:
> >
> >
> > Most of the existing XML tools are too CPU- and memory-intensive, I
> > agree, but NetBSD happens to be the center for improvements.
>
> have to edit the configuration files by hand. I would like to register,
> again, my strong objection to using XML -- or any other format which has
> XML's repugnant feature of token delimiters with semantic meaning, which
> is about as bad for human usability as possible -- in any application in
You will have to give an example of this "repugnant feature," because
your description does not ring a bell. Taking a wild guess, you're
stuck on some degenerate case involving entities.
> NetBSD in which it may be reasonably expected that humans will have to or
> choose to hand-edit the chosen data format.
I don't expect that at all. I wrote:
> I know that writing XML by hand is objectionable, so establish a
> one-to-one mapping from your user-friendly format to an XML format.
> Provide a tool [to convert] to/from the XML format.
Jason Thorpe
On Dec 14, 2009, at 4:40 PM, Thor Lancelot Simon wrote:
> Case 2: XML-like syntax -- with a huge artificial advantage because I've
> neatly indented it.
>
> <thing>
> <element-list>
> <element>1</element>
> <element>2</element>
> <element>3</element>
> <element>4</element>
> <element-list>
> <element>5</element>
> <element>6</element>
> </element-list>
> </element-list>
> </thing>
Of course, that is only one way to use XML. That could be written in a fashion somewhat easier on the eyes if your schema is designed for it:
<thing>
<element-list>
<element value="1"/>
<element value="2"/>
<element value="3"/>
<element value="4"/>
<element-list>
<element value="5"/>
<element value="6"/>
</element-list>
</element-list>
</thing>
-- thorpej
Marc Balmer
The main question is, who produces the configuration, and for what
purpose; and who uses it. If a human being is involved in this chain,
XML is not the right tool. If it is only machines, it might be the
right tool.
For the system installer, I assume that the configuration of what gets
installed, and how, is something a sysadmin wants to prepare. And
since sysadmins are half-human beings, using XML would mean torture.
Instead of splitting/rewritting/ sysinstall or replacing it with
bsdinstall, I'd suggest to evolve it: Add Lua for the configuration
and move the logic and user interface from C to Lua, keeping some core
functionality of an installer in C, but make it available to Lua. A
good installer would also let me install packages, maybe even meta-
packages, so that one of the first questions this installer asks, is:
"what kind of system do yo want to install?" and provides me with a
selection of meta-packages that are available on the distribution
medium (and for which the dependent packages are also available for
the architecture given).
Then the ideal installer would allow interactive operation or complete
unattended operation, where installation instructions are give in a
file.
Well, the ideal installer does not yet exist, but ahoka and myself
have some loose plans to make exactly this happen...
- Marc Balmer
Marc Balmer
Am 15.12.2009 um 16:03 schrieb Silas Silva:
> On Tue, Dec 15, 2009 at 08:12:45AM +0100, Marc Balmer wrote:
>> Instead of splitting/rewritting/ sysinstall or replacing it with
>> bsdinstall, I'd suggest to evolve it: Add Lua for the configuration
>> and move the logic and user interface from C to Lua, keeping some core
>> functionality of an installer in C, but make it available to Lua. A
>> good installer would also let me install packages, maybe even meta-
>> packages, so that one of the first questions this installer asks, is:
>> "what kind of system do yo want to install?" and provides me with a
>> selection of meta-packages that are available on the distribution
>> medium (and for which the dependent packages are also available for
>> the architecture given).
>>
>> Then the ideal installer would allow interactive operation or complete
>> unattended operation, where installation instructions are give in a
>> file.
>>
>> Well, the ideal installer does not yet exist, but ahoka and myself
>> have some loose plans to make exactly this happen...
>
> Hey Marc and all.
>
> Well, I was in doubt after reading the Turing-completeness problem of
> configuration vs. scripting and whether or not sysinst.conf should be
> Turing complete. But I mulled over it and I think using Lua on the
> project would be a good thing. One of the advantages of it is that we
> could trust the Lua parser, so it would not be necessary to create one
> more parser in Lex & Yacc as I previously suggested (although I'd love
> to do that :-P).
>
> Bringing Lua wields a bigger question: shouldn't NetBSD import to its
> tree, a powerful general-purpose interpreter? (sh, ksh, awk, bc, etc.,
> are not general-purpose). Why not just import Lua to the base system so
> people can built projects on it, like sysinst? Just an idea.
I am working on this. Myself and some other developers want Lua in base. For different projects, actually. There is an email thread on tech-userlevel about this.
>
> I've already used Lua embedded in a C++ program, not too much, but in a
> level enough to note that it works great and glue very well.
>
> Your ideas for sysinst are great. I would love to help on this project!
>
> Well, I'm not going to repost all the XML pro and cons that people
> posted here, but I disagree with the adoption of XML. However, since I'm
> much worried about helping bringing a new sysinst to live, I'll work on
> any solution, whatsoever.
The ones that will do the work will choose the tools, not the bystanders...
>
> Couldn't sysinst import/export from/to both formats, human-readable (Lua
> script or another) and XML? That would solve the controversy, but it may
> be difficult to maintain support for two different standards.
What would the gain be of having two configuration "languages"? I guess it ould only add to complexity.
>
> Thank you very much!
>
> --
> Silas Silva
Silas Silva
> bsdinstall, I'd suggest to evolve it: Add Lua for the configuration
> and move the logic and user interface from C to Lua, keeping some core
> functionality of an installer in C, but make it available to Lua. A
> good installer would also let me install packages, maybe even meta-
> packages, so that one of the first questions this installer asks, is:
> "what kind of system do yo want to install?" and provides me with a
> selection of meta-packages that are available on the distribution
> medium (and for which the dependent packages are also available for
> the architecture given).
>
> Then the ideal installer would allow interactive operation or complete
> unattended operation, where installation instructions are give in a
> file.
>
> Well, the ideal installer does not yet exist, but ahoka and myself
> have some loose plans to make exactly this happen...
Hey Marc and all.
Well, I was in doubt after reading the Turing-completeness problem of
configuration vs. scripting and whether or not sysinst.conf should be
Turing complete. But I mulled over it and I think using Lua on the
project would be a good thing. One of the advantages of it is that we
could trust the Lua parser, so it would not be necessary to create one
more parser in Lex & Yacc as I previously suggested (although I'd love
to do that :-P).
Bringing Lua wields a bigger question: shouldn't NetBSD import to its
tree, a powerful general-purpose interpreter? (sh, ksh, awk, bc, etc.,
are not general-purpose). Why not just import Lua to the base system so
people can built projects on it, like sysinst? Just an idea.
I've already used Lua embedded in a C++ program, not too much, but in a
level enough to note that it works great and glue very well.
Your ideas for sysinst are great. I would love to help on this project!
Well, I'm not going to repost all the XML pro and cons that people
posted here, but I disagree with the adoption of XML. However, since I'm
much worried about helping bringing a new sysinst to live, I'll work on
any solution, whatsoever.
Couldn't sysinst import/export from/to both formats, human-readable (Lua
script or another) and XML? That would solve the controversy, but it may
be difficult to maintain support for two different standards.
Thank you very much!
--
Silas Silva
--
Greg A. Woods
Subject: Re: sysinst split project - The Configuration File
>
> *scripting*. Configuration should not be Turing-complete, because it
> becomes unmanageable. (See "sendmail".) Scripting is Turing-complete
> by definition.
I'm not so sure about that.
For one I think your argument using Sendmail as an example is more or
less worthless.
Sendmail configurations essentially describe a state machine -- and they
were not, at least when they gained their infamy, written in a truly
Turing Complete programming language. The real problem with Sendmail
configurations is that they are written in a language that is worse
(i.e. more terse, convoluted and more difficult to understand) than Perl.
> 3. Configuration should be done in a concise and legible format,
> such as key/value pairs, nested key/value pairs, columnar tables, or
> whatever. Since XML is content-free^W^W just syntax,
Indeed, on both points! :-)
> any such sane
> format can easily be represented as XML.
Well, perhaps, but then you necessarily throw away the "legible format"
part again.
> 4. Parsers are cheap.
Sometimes they are not -- or rather there are scales of complexity and
overhead and other such costs in parsers, especially depending on the
language syntax they are intended to work with.
> If you have both configuration and scripting, the configuration data
> should be made available (as data, in a clearly defined schema) to the
> user scripts.
Indeed. Typically in my experience this is easiest done, and perhaps
more importantly most effective, if the configuration language is the
same one as is used to define data structures and values in the
scripting language of choice, if at all possible. But then I am very
much a lisp fan. :-)
Jason Thorpe
On Dec 14, 2009, at 11:24 PM, David Holland wrote:
> 3. Configuration should be done in a concise and legible format,
> such as key/value pairs, nested key/value pairs, columnar tables, or
> whatever. Since XML is content-free^W^W just syntax, any such sane
> format can easily be represented as XML. (As long as the data remains
> strictly hierarchical and isn't, say, graph-structured. But that's not
> likely to be a problem here.)
Right. I tend to favor property lists for this ... but the prevalent format for those in modern times are the XML flavor.
-- thorpej
Silas Silva
> I am working on this. Myself and some other developers want Lua in
> base. For different projects, actually. There is an email thread on
> tech-userlevel about this.
I just finished reading that thread. sysinst implemented in Lua looks
like a great example for the chicken and egg problem. People asked for a
list of uses? Here we get one.
> The ones that will do the work will choose the tools, not the
> bystanders...
You are right :-)
> > Couldn't sysinst import/export from/to both formats, human-readable
> > (Lua script or another) and XML? That would solve the controversy,
> > but it may be difficult to maintain support for two different
> > standards.
>
> What would the gain be of having two configuration "languages"? I
> guess it ould only add to complexity.
IMO, there would be no gain. But, if libprop were used, it would take
just some more lines of code to import/export them from/to Property
Lists, which are XML.
Ok, proplib(3) says proplib parser is not a read XML parser, but it is
almost one.
--
Silas Silva
Jason Thorpe
On Dec 15, 2009, at 3:40 PM, George Michaelson wrote:
> I use OSX as my day-job computer, with NetBSD at home on my raidframe filestore, and on virtualboxen.
>
> I'd be quite happy with an XML based configuration file methodology.
>
> After all, OSX uses .plist files. -So, I'm already 'there' for some value of there.
Right, and NetBSD's proplib is compatible with Apple XML plist files. And a bonus for the XML haters out there... it's not even a full-blown XML reader ... proplib is hard-coded for XML plists, so no need to pull in an XML library or even the DTD.
> Complexity of editing? phui. Please. Don't try my patience. vi/emacs/ed and grep and sed and now shell level XML tools to do grep? I'm happy.
>
> I'll trade size for consistency any time. the files are huge? they're still smaller than boot-time VESA logo .GIF images!
>
> To the lua fans. Look, I don't disbelieve you that this can work, or other OS do this. But, I'm struggling to see this fitting naturally into a BSD community framework. OpenBSD and FreeBSD and Dragonfly are not just 'interesting' -there is active codeshare. I worry that this step is a step which winds up reducing mindshare, and codeshare, irreversably.
>
> So, as a (l)user not a developer, please, leave lua on the pkgsrc side, for now. Do XML because its a small, natural step, and it looks to me that in the BSD mindspace, its already 'there' -in OSX
To be fair ... OS X does not have any BSD-level APIs for manipulating plists (although there are command line tools that consume/produce them)... the lowest-level API available for those is CF.
>
> -George
David Holland
> > 1. Most of the rest of this thread is conflating *configuration* and
> > *scripting*. Configuration should not be Turing-complete, because it
> > becomes unmanageable. (See "sendmail".) Scripting is Turing-complete
> > by definition.
>
> I'm not so sure about that.
>
> For one I think your argument using Sendmail as an example is more or
> less worthless.
sendmail is only the most glaring example.
> Sendmail configurations essentially describe a state machine -- and they
> were not, at least when they gained their infamy, written in a truly
> Turing Complete programming language.
Oh? The Towers of Hanoi thing was just a joke?
The basic reason configuration, which shouhld be data, should not be
expressed in a Turing-complete way is that doing so makes it nearly
impossible to write correctly-functioning configuration management
tools.
As in, you can have tools that select among canned configurations, but
not tools that can interpret and edit an existing configuration.
> > 3. Configuration should be done in a concise and legible format,
> > such as key/value pairs, nested key/value pairs, columnar tables, or
> > whatever. Since XML is content-free^W^W just syntax,
>
> Indeed, on both points! :-)
>
> > any such sane
> > format can easily be represented as XML.
>
> Well, perhaps, but then you necessarily throw away the "legible format"
> part again.
Yes, but that's not the point; the point is that people who want XML
for some reason can get it via an automatic conversion step.
> > 4. Parsers are cheap.
>
> Sometimes they are not -- or rather there are scales of complexity and
> overhead and other such costs in parsers, especially depending on the
> language syntax they are intended to work with.
Parsers for anything suitable for representing configuration data are
cheap. For that matter, even parsers for sane programming languages
are cheap. Parsing C++ is not cheap, but that's a whole different
problem.
> > If you have both configuration and scripting, the configuration data
> > should be made available (as data, in a clearly defined schema) to the
> > user scripts.
>
> Indeed. Typically in my experience this is easiest done, and perhaps
> more importantly most effective, if the configuration language is the
> same one as is used to define data structures and values in the
> scripting language of choice, if at all possible.
This is exactly the model I was arguing against. Or one of them, anyway.
> But then I am very much a lisp fan. :-)
no comment.
--
David A. Holland
dhol...@netbsd.org
David Holland
> > 3. Configuration should be done in a concise and legible format,
> > such as key/value pairs, nested key/value pairs, columnar tables, or
> > whatever. Since XML is content-free^W^W just syntax, any such sane
> > format can easily be represented as XML. (As long as the data remains
> > strictly hierarchical and isn't, say, graph-structured. But that's not
> > likely to be a problem here.)
>
> Right. I tend to favor property lists for this ... but the
> prevalent format for those in modern times are the XML flavor.
Config files meant for manual editing should use some other syntax.
Why is it again that proplib can't handle a simple key=value and
key={ key=value... } syntax?
--
David A. Holland
dhol...@netbsd.org
--
David Holland
> Well, I was in doubt after reading the Turing-completeness problem of
> configuration vs. scripting and whether or not sysinst.conf should be
> Turing complete. But I mulled over it and I think using Lua on the
> project would be a good thing.
It seems to me that there are at least three different questions here.
One is: whether canned answers to canned questions (like "which
password cipher?" or "what IP address do I use?") should be expressed
as static data or as script hooks.
The second is: whether it should be possible to direct the overall
progress of the install process using a user-supplied script, which
e.g. plugs together components of sysinst and executes them.
The third is: whether pieces of sysinst itself should be written as
scripts, and if so, whether part of the available configuration
process should involve editing those scripts.
ISTM that canned answers should be expressed as canned data. (Or
possibly via some non-Turing-complete indirect map, e.g. based on the
hostname or IP address. But even then I think it'd probably be better
to map from hostnames to complete install profiles rather than retype
the map for every install question.)
It also seems to me though that directing the overall progress of the
install is useful and perhaps important, and that should be done in a
script language. Striking the right balance between this and requiring
the user to edit complex scripts that ought to be internal to sysinst
will take some thought.
> Couldn't sysinst import/export from/to both formats, human-readable (Lua
> script or another) and XML? That would solve the controversy, but it may
> be difficult to maintain support for two different standards.
Trying to transform scripts into XML is a bad idea.
--
David A. Holland
dhol...@netbsd.org
--
David Holland
> It also seems to me though that directing the overall progress of the
> install is useful and perhaps important, and that should be done in a
> script language. Striking the right balance between this and requiring
> the user to edit complex scripts that ought to be internal to sysinst
> will take some thought.
The other thing that's been missing from this discussion, btw, is an
analysis of the strengths and flaws of other projects' tools for doing
automatic installs...
--
David A. Holland
dhol...@netbsd.org
--
Greg A. Woods
Subject: Re: sysinst split project - The Configuration File
>
>
> One is: whether canned answers to canned questions (like "which
> password cipher?" or "what IP address do I use?") should be expressed
> as static data or as script hooks.
>
> The second is: whether it should be possible to direct the overall
> progress of the install process using a user-supplied script, which
> e.g. plugs together components of sysinst and executes them.
>
> The third is: whether pieces of sysinst itself should be written as
> scripts, and if so, whether part of the available configuration
> process should involve editing those scripts.
>
> install is useful and perhaps important, and that should be done in a
> script language. Striking the right balance between this and requiring
> the user to edit complex scripts that ought to be internal to sysinst
> will take some thought.
I think you've hit the nail truly on the head there!
David Young
> Use XML for the configuration. That ensures that the configuration is
> writing XML by hand is objectionable, so establish a one-to-one mapping
> the XML format.
I think that this point is important, but it seems to have caused a lot
of misunderstanding and FUD, so I am going to try again.
If NetBSD puts structured[1] configuration files into a common
format[2], then it begins to make sense to create the tools for the
base system that can process that format in pipelines and in scripts.
If we keep adding an ad hoc configuration format for every new
application, like the bad old ways of UNIX, then it is not useful
to write even a single tool to process anything.
It does not matter as much whether the common format is JSON, YAML,
XML, or something else, as it matters that there is one format to
process. However, I recommend making XML the common format because so
many programs and services already generate so much useful XML, because
memory- and CPU-efficient tools for processing it in the UNIX idiom are
in the works for NetBSD, and because XML, in the form of property lists,
already has a lead over other formats in NetBSD.
Dave
[1] By "structured," I mean formats that come in sections (think .ini
files), that are "hierarchical" or have "scopes", as opposed to
tabular formats with columns delimited by whitespace or some other
reserved character class.
[2] "Format" or "meta-format" if you prefer.
--
David Young OJC Technologies
dyo...@ojctech.com Urbana, IL * (217) 278-3933
--
Aaron J. Grier
> Jokes aside, I agree with pros of the common format but I don't think
> XML is a good choice.
XML, proplists, json, lua -- the actual file format is an implementation
detail that the implementor gets to decide.
so long as configuration files and installation scripts are losslessly
transformable between multiple formats, everybody wins.
--
Aaron J. Grier | "Not your ordinary poofy goof." | agr...@poofygoof.com