For Review: document layout change idea and beginging of more detailed distro specific instructions

18 views
Skip to first unread message

Matt Darcy

unread,
Jul 28, 2016, 7:32:42 AM7/28/16
to vexim
All,

based on the mess I was making editing the README.md in the current 2.3 branch, I sat and started an idea I'd wanted to do with vexim2 for ages but never really put the time in to get it moving.

I've changed the layout of the documentation and installation instructions in my personal branch https://github.com/ikonia/vexim2

I've put a simplified README.md in place to introduce you to vexim and what you need.
I've renamed the "docs" directory to be "examples" as they are config examples
i've created an "install-guides" directory that is referenced from the README.md  which now contains example instructions of how to install vexim2 on specific distributions.

I think this will make it easier to maintain than a big README as.

a.) it's clear what guide you should use
b.) you can put any distro specific work arounds (I'm finding a few changes needed for Centos/RHEL 6 and Ubuntu 16.04) in the distro specific file.
c.) it's easier to maintain as it's clear what file your editing for what distro - and you can put a lot more detail in without it getting cluttered/long file (eg: stop referencing generic locations like /usr/local/exim - which no distro uses, and start referencing the proper distro locations)
d.) you can add to this later if you need - if a new distro has it's own way of doing it you can create a distro file without having to navigate through a big README with different sections to work out what you need.
e.) you could also split this later to be version specific, eg: Debian-8, Debian-7, Debian-9 if they have differences.


These are FAR from complete - it's more to get feedback on the new layout and the level of detail I'm putting into the specific distro install guides. A lot is reusing the excellent info from the old README.md files, and just adding wording and additional details.

I thought I'd get feedback now before progressing any further with this to see if it was well received, if it is I'll continue this to completion and submit a big merge request for all the documentation, if not, I'll stop now and not waste any more time.

If you want to contribute to the branch I'll give write access to you, especially if you have distro specific knowledge. This should allow a comprehensive and easy to maintain set of docs to be built up for the 2.3 branch.

Thoughts and opinions really appreciated.

Matt

Matt Darcy

unread,
Aug 2, 2016, 6:31:58 AM8/2/16
to vexim
No feedback on this ? is it worth me continuing with this or just dump it ?

Udera

unread,
Aug 2, 2016, 7:39:26 AM8/2/16
to Matt Darcy, vexim
Sorry for our silence. I was starting to run a few tests on FreeBSD
(which is new to me).

On 2016-08-02 12:31, Matt Darcy wrote:
>> I've put a simplified README.md in place to introduce you to vexim
>> and what you need.
>> I've renamed the "docs" directory to be "examples" as they are
>> config examples
>> i've created an "install-guides" directory that is referenced from
>> the README.md which now contains example instructions of how to
>> install vexim2 on specific distributions.
Currently the README.md is a bit of a mess. There are obviously some
thing out of date that need to be fixed. Up to now it was considered to
be a linear installation instructions, so you could literally print it
out and work through it. A nice way to clearly differentiate the parts
for different systems would be great. Dedicated files for each system
would be clearer but there is a lot of redundant information that must
be updated when something changes.

>> c.) it's easier to maintain as it's clear what file your editing for
>> what distro - and you can put a lot more detail in without it
>> getting cluttered/long file (eg: stop referencing generic locations
>> like /usr/local/exim - which no distro uses, and start referencing
>> the proper distro locations)
The install guide was written for FreeBSD system, the file system is
organized differently from Linux systems (which are quite similar). So
these strange paths are still used.

Perhaps we can reduce redundancy, when we keep a central installation
guide which is then reduced to the essential information. The
distribution-specific commands how to install your webserver with php
and mysql can be put in a different file. Some with a working webserver
can then easily skip this step.

>>
>> These are FAR from complete - it's more to get feedback on the new
>> layout and the level of detail I'm putting into the specific distro
>> install guides. A lot is reusing the excellent info from the old
>> README.md files, and just adding wording and additional details.
True. Especially the setup of clamav and spamassassin is missing. There
was a very complete installation guide for debian systems (in German
unfortunately).

Just another idea of the configuration file: We keep the current
configuration file, where a lot of alternative paths are mentioned. For
the vexim2.3-release we create a branch and then prepare a specific
exim.conf for debian, centOS and FreeBSD. We test it now, but this way
we are not required to maintain different sets of configuration files.

Thanks for your efforts,
Udera

Matt Darcy

unread,
Aug 2, 2016, 8:06:08 AM8/2/16
to vexim
Thanks for the feedback it's all good and valid.

I'll respond in line, see what you think.



>> Currently the README.md is a bit of a mess. There are obviously some
>> thing out of date that need to be fixed. Up to now it was considered to
>> be a linear installation instructions, so you could literally print it
>> out and work through it. A nice way to clearly differentiate the parts
>> for different systems would be great. Dedicated files for each system
>> would be clearer but there is a lot of redundant information that must
>> be updated when something changes.

I think this is mostly true, but doesn't show up in my small test example commit. The amount of "generic" information shared between install methods isn't actually that much, there are some obvious things, such as executing the mysql database creation commands (easy example) but most of the rest will be distro specific. The concepts will be the same, eg: copy file A to location B - but that location will be different on a distro by distro basis. Certainly small enough differences for an initial release to have a full install guide for each major distro (including BSD) manually created as a starting point. There is more to this that I'll explain in more detail as it's more relevant there. I freebsd paths should still be used, but only in the freebsd guide.







>> The install guide was written for FreeBSD system, the file system is
>> organized differently from Linux systems (which are quite similar). So
>> these strange paths are still used.

>> Perhaps we can reduce redundancy, when we keep a central installation
>> guide which is then reduced to the essential information. The
>> distribution-specific commands how to install your webserver with php
>> and mysql can be put in a different file. Some with a working webserver
>> can then easily skip this step.

as above the duplicate information in detail is not actually that much, especially as you say around freebsd certainly for an initial release splitting this out into clear install guides "per distro, per file" would have some duplication, certainly around the high level process, but the details are actually mostly unique.


>> True. Especially the setup of clamav and spamassassin is missing. There
>> was a very complete installation guide for debian systems (in German
>> unfortunately).

>> Just another idea of the configuration file: We keep the current
>> configuration file, where a lot of alternative paths are mentioned. For
>> the vexim2.3-release we create a branch and then prepare a specific
>> exim.conf for debian, centOS and FreeBSD. We test it now, but this way
>> we are not required to maintain different sets of configuration files.

Splitting the install methods into different distro files (and as you say different example config files in the example directory) makes this a lot easier to setup a complete guide process even for more advanced features such as spamassassin.

for the 2.3 initial release - this work will not be done - thats why I was maintaining this on my own branch it was all complete - and maybe 2.3.1 patch release could include a big documentation upgrade such as the example I'm putting together here. I'd be happy to give write access to my branch to get the documentation polished by people with specific experience.

The Debian work that was already done was very good, and when I started writing this out into the distro specific "debian" guide, I found it a really clear and easy process, much clearer than trying to follow the current README.md and pick out the bits I wanted to do and bits I wanted to ignore. I've been testing the documentation (so far) with a collegue who isn't the the strongest user and getting feedback on what's clear and what needs to be improved.

To answer the question raised earlier about duplicate work, my plan once we had a working set of documentation to act as a template, was to convert the documentation to a latex style template, that way the documentation is automatically generated and duplication is a single change.

eg:

global --

** mysql ***

mysql -u root < exim_mysql.sql


and then each document, debian, centos, freebsd etc etc, includes ** mysql ** so that when we change

** mysql **

mysql -u root -h localhost < exim_mysql.sql

all the other documents pickup this change and include the change.

It will make changes easy to add/remove sections, and keep generic documentation up to date, with only very distro specific changes needing to be manually edited on a per distro/file basis.

The same could be done with example configuration files, using Ruby ERB's (for example) so that the exim.conf is always the same, but the distro specific parameters will be included/excluded/set depending on the parameter,

eg:

exim.conf snippet

# The following ACL entry is used if you want to do content scanning with the
# exiscan-acl patch. When you uncomment this line, you must also review the
# acl_check_content entry in the ACL section further below.

acl_smtp_data = <= @acl_check_content =>

# This ACL added by Avleen Vig will drop all mail where the sender's helo
# command contains your IP address. Please check the IP address at the top
# for the configure file to make it your own.

acl_smtp_helo = <= @acl_check_helo =>

<= if OS == "centos" > {
# This ACL can be used to refuse certain file extension in ZIP files

#acl_smtp_mime = acl_check_mime
}

(just an really small snippet example)

so again only the templates are maintained - but can be used to generate say 5 different variations.

I've had really positive results on this working on the linux from scratch book, the whole book / instructions is latex generated, and after you've got the template setup, the changes and maintenance is really really easy.

To do this though, you need a working template, hence why say 2.3.1 have the new documentation layout in place with working documentation and then 2.3.2 be a maintenance release that sets up the templates and sets up a structure that could be taken forward into vexim 3 for example.


thoughts ?





On Thursday, 28 July 2016 12:32:42 UTC+1, Matt Darcy wrote:

Rimas Kudelis

unread,
Aug 3, 2016, 8:07:25 AM8/3/16
to ve...@googlegroups.com
Hi Matt and all,

I won't be replying to any particular points in your e-mail, but more to the overall idea.

Like @Udera said, the guide (current README.md) was probably initially meant as a print-out doc, which you would just follow. And I think it was initially targeting FreeBSD only, as that's what Avleen used at the time.

Then I added some Debian-specific things to it.

Then we figured out "hey, we could add even more distro/os-specifics into it", and in the end, we have this messy mess that we're all struggling to improve right now.

I guess it's quite obvious that trying to make a manual for everybody by filling it with all the various steps for each and every situation didn't really work out well.

But looking at our documentation right now, I think we can still try to improve it in different ways:
  1. First of all, I think we should strive to reduce the number of steps we describe to a bare minimum. If editing something is optional, or depends on a condition of some sorts, it should probably not live in the master manual. Furthermore, if something can be done by us outside the manual, it should be done that way.
  2. Related to above: I think the values specified in variables.php should be documented in that file, not in the manual. The manual should defer to that file instead. This would help us avoid inconsistencies between code and the manual (there already are some).
  3. We should probably leave out distro specifics, at least for the distros/systems we aren't usually using ourselves and for which have no volunteer to maintain these notes. If we're ending up setting up virtual machines with the OS we aren't familiar with just to improve the manual, is it worth the effort?
  4. Exim configuration: I'm strongly in favor of moving that part under docs/. In my opinion, it's just another software we can hook into. We don't have a manual for Postfix or any other MTA yet (if I remember correctly), but that doesn't mean that it's impossible to write one. Befriending Courier MTA might be particularly easy, since we already have manuals for their IMAP and POP3 servers.
  5. Related to above: I'm actually a bit unsure how to treat all these manuals for various third party servers. When should we consider a manual relevant and when not?  For example, we have decade-old manuals for Qpoppoer and tpop3d. These deamons seem quite obsolete today. Should we drop these manuals, move them to some "archive", or what? Also related: should we bother maintaining manuals for different versions of Vexim and software in question, or should we always target latest Vexim and a few latest releases of the software in question? I'm leaning towards the latter, but then I guess GitHub Wiki might not be the best platform for our documentation and we should leave it in docs/ instead, so that at least in theory the state of docs could always reflect the state of code in the same branch.

Also, not strictly on topic, but I think we should investigate the possibility of generating/providing packages for different operating system, and letting the packaging system do at least some of the work. Having packages would let us customize various aspects of the configuration (such as paths, dependencies, user creation) for target operating systems, thus again shifting this burden from the end users' shoulders.

Rimas

Udera

unread,
Aug 3, 2016, 8:37:38 AM8/3/16
to Rimas Kudelis, ve...@googlegroups.com
On 2016-08-03 14:07, Rimas Kudelis wrote:
> * Related to above: I'm actually a bit unsure how to treat all these
> manuals for various third party servers. When should we consider a
> manual relevant and when not? For example, we have decade-old manuals
> for Qpoppoer and tpop3d. These deamons seem quite obsolete today.
> Should we drop these manuals, move them to some "archive", or what?

Courier and Dovecot provide a full IMAP server and support the
POP-protocol as well. This should match most people needs and as we are
using them, we can provide configuration examples.

Qpopper is no longer maintained (https://en.wikipedia.org/wiki/Qpopper)
and tpop3d neither (or it's very quiet:
http://savannah.nongnu.org/projects/tpop3d/), so let's drop them. I
suppose no one is willing to test them. Someone who wants to use a
different IMAP or POP server can ask on the mailing list for help.

Matt Darcy

unread,
Aug 3, 2016, 9:29:17 AM8/3/16
to vexim
some solid points on there - some I don't agree with.

I can see the history of where this has come from and how it has evolved to what it is now, while it functions now - I don't think it's something we should take forward and improve. The single file setup is just not fit for purposes, and I can't find any other projects I use that use this approach. I think it should just be stopped and work put into a better more modular layout.

I don't think there are many steps in place now - so I'm not sure what there is to strip back on, it's quite a simple process, the more complicated items are the contents of the example files and making them work with your distribution. When I started the document layout I'm trying at the moment I made a crip note of the install process and it was really only 8 simple steps, end to end.
I'm sure one or two could be removed or simplified.

While improving comments in the example config files is great, configuring a file from comments is not good as it just tells you what to populate, not what it is, and how it's used (generally). Having details on what you are changing and why in external documentation is important. That doesn't mean more and better comments in the files are not good, I just don't think relying on comments is good documentation.

I don't think using other MTA's is something thats worth while even discussing - the product is valled vexim as it's designed to work with exim, there are other options for postfix and other mta's if you want vexim3 to be plug and play for other mta's thats a different discussion for for v2.X - I don't see any point in even discussing it.

I feel very strongly that distro specific config should not be left out - even if for the initial release we have to put in work to get a valid working document, if people don't see instructions for how to use it on their system they won't look at it, if we want people with experience to help - it's easier to get them to help with existing content there rather than saying start from scratch. That doesn't mean document all distros, but certainly the main ones with differences, eg: RHEL/Scientific/CentOS are all the same, so enterprise linux covers that, Debian 7/8 would be the same, Ubuntu 14.04 and ubuntu 16.04 would be different. A BSD and maybe Suse would cover the whole platform set. That would be for me a good target for the initial documentation upgrade.

things like imap and pop3 servers I think you have to be pragmetic, look at the main ones, look at what the distros are shipping and just look at those.

In terms of packages, I already have Exim packages for RHEL and base packages for Debian (could probably port to ubuntu easy enough) - this will remove some of the distro specific steps manually (should still be documented) if you remember I flagged this as something I wanted to do a long time ago, but we need working system configs for stable OS platforms. I'd be happy to progress this along with documentation, as like you I feel this is important.

agreeing a set of platforms and products on those platforms we want to support and putting them on the wiki would be a good start, I'd be happy to put the effort into this, but I need to know where I'm going and how I'm doing it is what others want and like, so I don't waste time and effort.

Matt

Rimas Kudelis

unread,
Aug 4, 2016, 3:46:35 AM8/4/16
to ve...@googlegroups.com
Hi,

2016-08-03 16:29, Matt Darcy wrote:
> some solid points on there - some I don't agree with.
>
> I can see the history of where this has come from and how it has
> evolved to what it is now, while it functions now - I don't think it's
> something we should take forward and improve. The single file setup is
> just not fit for purposes, and I can't find any other projects I use
> that use this approach. I think it should just be stopped and work put
> into a better more modular layout.

I think you should try it out and see what you get. It's hard to judge
the worthiness of separating these manuals by distribution without
seeing an example, and so far, your EL manual is just an empty file.

> I don't think there are many steps in place now - so I'm not sure what
> there is to strip back on, it's quite a simple process, the more
> complicated items are the contents of the example files and making
> them work with your distribution. When I started the document layout
> I'm trying at the moment I made a crip note of the install process and
> it was really only 8 simple steps, end to end.
> I'm sure one or two could be removed or simplified.

Exactly.
For example, I've already removed the need to edit the mysql.sql file
before installing it. I think the useradd/adduser command might be
unified (or almost unified). The Perl script is now obsolete. What
remains is mostly the Exim configuration file editing, which, I think,
can be mostly documented in the file itself. Or not.

> While improving comments in the example config files is great,
> configuring a file from comments is not good as it just tells you what
> to populate, not what it is, and how it's used (generally). Having
> details on what you are changing and why in external documentation is
> important. That doesn't mean more and better comments in the files are
> not good, I just don't think relying on comments is good documentation.

Well there's no limit to the size and meaning of comments you can put in
the config file, so it's not like you *cannot* document the file in
itself, it just might prove impractical. OTOH, if we could balance
high-level overview in the "manual" and lower level explanations in the
config file, this might probably work. I'm not saying it definitely
will, just theorising.

> I don't think using other MTA's is something thats worth while even
> discussing - the product is valled vexim as it's designed to work with
> exim, there are other options for postfix and other mta's if you want
> vexim3 to be plug and play for other mta's thats a different
> discussion for for v2.X - I don't see any point in even discussing it.

Vexim was designed to work with pretty much anything you could throw at
it. Exim was used as the model, and our Exim integration (in the form of
config files) is the most complete at the moment, but there is
absolutely *nothing* in Vexim that depends on Exim in any way. I don't
think it could even be othewise, since our PHP code is not hooking into
the MTA – there's no need for that. So, in my opinion, we could provide
instructions for Exim, Postfix etc. the same way we provide them for
Courier-IMAP, Dovecot etc. It's not like our PHP frontend cares about
what backends are using its database.

> I feel very strongly that distro specific config should not be left
> out - even if for the initial release we have to put in work to get a
> valid working document, if people don't see instructions for how to
> use it on their system they won't look at it, if we want people with
> experience to help - it's easier to get them to help with existing
> content there rather than saying start from scratch. That doesn't mean
> document all distros, but certainly the main ones with differences,
> eg: RHEL/Scientific/CentOS are all the same, so enterprise linux
> covers that, Debian 7/8 would be the same, Ubuntu 14.04 and ubuntu
> 16.04 would be different. A BSD and maybe Suse would cover the whole
> platform set. That would be for me a good target for the initial
> documentation upgrade.

Debian and Ubuntu are likely to turn out sharing the same manual. The
packages in these are, or at least used to be, same. Paths even more so.

I'm not saying we should avoid these specific instructions. I just don't
think we should prioritize them to the level of actually spending time
doing them without seeing demand for them first. I mean, it's fun to
spend time doing that (I'm actually prone to wasting time on things like
that, just to satisfy my curiosity), but without real demand, this might
turn out being wasted effort.

> things like imap and pop3 servers I think you have to be pragmetic,
> look at the main ones, look at what the distros are shipping and just
> look at those.

I'd say same about MTA's as well. :) Then again, it's not like we should
force ourselves to write manuals for other MTA's. If anyone wants –
welcome, if not – Exim alone will do for now.

> In terms of packages, I already have Exim packages for RHEL and base
> packages for Debian (could probably port to ubuntu easy enough) - this
> will remove some of the distro specific steps manually (should still
> be documented) if you remember I flagged this as something I wanted to
> do a long time ago, but we need working system configs for stable OS
> platforms. I'd be happy to progress this along with documentation, as
> like you I feel this is important.

I was talking about packaging Vexim, not about Exim packages.

> agreeing a set of platforms and products on those platforms we want to
> support and putting them on the wiki would be a good start,

I think it's difficult to promise anyone that we'll support platforms A,
B and C and products D, E and F. We're an open-source project with just
a handful of developers with little time on their hand. We may list what
we support, but I don't want this to become a burden, or a release
blocker. On the other hand, I don't mind seeing users asking for help,
even if they would use an "unsupported" (by us) OS or product.

> I'd be happy to put the effort into this, but I need to know where I'm
> going and how I'm doing it is what others want and like, so I don't
> waste time and effort.

Well again, I'd say go ahead and adapt a manual for EL (also please
rename them from .txt to .md, so that GitHub would treat them as
markdown files). Like I said above, it's somewhat difficult to judge
usefulness of separate OS-specific manuals without having at least a
couple of them to look at.

Cheers,
Rimas

Matt Darcy

unread,
Aug 4, 2016, 6:54:06 AM8/4/16
to vexim
Wise words,

I've got the Debian install almost completed and I've got notes on my EL install, once it's %101 working I'll put that into a document.

I'm not sure ubuntu and debian (certainly ubuntu 16.04) may always be able to share the same install as ubuntu do make some modifications, but we can work that out later.

I did mean to type "vexim" packages, I've got a package set for EL and Debian already both could use a little tidy up and some fixes, but thats doable, and this may make a huge part of the install instructions redundant anyway.

Lets see if I can get this done quickly, at least a draft and see how it looks.

Matt


On Thursday, 28 July 2016 12:32:42 UTC+1, Matt Darcy wrote:
Reply all
Reply to author
Forward
0 new messages