Some large diffs after 10.48

[Nicholas Wilson]

Hello everyone,

 

After the release of 10.48 I feel that I’d like to make a few slightly disruptive changes, while everything is peaceful.

 

I have three things which I have been putting off, because it didn’t feel like the right moment.

 

Firstly, I am going to delete the old Windows RunTest.bat, RunGrepTest.bat files. These have been a real pain to maintain, because the Windows bat quoting rules keep causing trouble; the syntax is hard to test; and newline issues make it quite brittle and hard to keep in sync with the Unix shell scripts.

 

As a replacement, I have created RunTest.py and RunGrepTest.py. These are cross-platform and identical in behaviour to the shell scripts. They must be used on Windows, and can optionally be used on other platforms. I am not proposing (at this time) to remove the shell scripts and force all Unix/Mac/Linux clients to use Python.

 

The main downside is that Windows clients now need to have Python installed – unless of course they turn off the tests (with PCRE2_BUILD_TESTS=OFF in CMake). This does not seem like a very difficult requirement to impose on consumers. (In fact - requiring Python for the Unix test suite would be quite normal in many projects.)

 

In the future, having a Python test harness could allow us to do more flexible testing, with a wider choice of test suites, and libraries such as fuzzers or other pieces of test tooling.

 

Secondly, I plan to reformat the codebase. I know it’s silly! However, the last time we discussed it, I think we agreed that a slightly more contemporary style could be more welcoming for new contributors. I will do this in two passes. I have a short Clang-format config for ‘regularising’ the current formatting. It affects fewer than 10% of the lines of code, but leaves the current bracing style intact. Because the changes don’t affect all the lines, I can read the diff and ensure that it’s not too ugly.

 

The second pass will then be a mass re-indentation of braces. I have a small and stupid Python script for that, which mechanically adjusts lines that start with whitespace followed by brace, and also recognises ‘case…:’ and ‘default:’ at line start.

 

The end result will match my (our) Clang-format style exactly.

 

I hope to maintain the character and overall style of the code. The bulk change is simply moving the braces around.

 

Thirdly, I intend to reformat the manpage documentation as Asciidoc source. This is a format which is simpler than Sphinx/reStructuredText, and richer than Markdown. I will start by taking each manpage, and turning it into an Asciidoc file (one file to one file). The words and content will not change.

 

Asciidoc will then be the primary source, and we will use its CLI tool to derive the HTML and manpage output. The actual files we ship should be nearly identical to the current ones. Hopefully, some of the formatting may be a little prettier (and certainly not uglier).

 

Asciidoc is used to generate the ‘man git’ manpages, and the Git official website, as well as other well-known OSS projects.

 

In the future, this should let us write new documentation much more easily, and reorganise the website with more content.

 

It is hard to write a modern website that’s well-formatted and easy to navigate, using the roff/man syntax.

 

In conclusion, all these changes are relatively mechanical, and tool-driven. It’s not an AI rewrite. However, it will generate some very large diffs.

 

Please do let me know any feedback you have, or preferences.

 

I am likely to do all three of these, unless someone has a strong preference or convinces me otherwise.

 

All the best,

Nick

[Philip Hazel]

Hello all,

Here are some comments from a historical point of view. Executive summary: a spring clean every now and again is a good idea, and PCRE is nearly 30 years old now.

1. Converting RunTest to Python: When I first wrote PCRE it was an 8-bit only, ASCII-only, program. I hacked up a quick pcretest program with a few single-letter options. Later came RunTest, which didn't have much to do at first. For PCRE2, there was a complete redesign for pcre2test (thanks to a prod from Zoltan) but RunTest remained a shell script. I guess if I'd seen the future, I would have used something else from the start - probably Perl, because that was what I knew (it being older than Python). So yes, I think upgrading to RunTest.py is probably a good idea. 

2. I adopted my indenting and bracing style around 1968 (writing BCPL, before C was around) and I know it is not what most people use.  Making the code more readable for people is a good idea.

3. About 20 years ago, as I was tidying the Exim documentation process, I flirted with Asciidoc for a couple of releases. This is a comment from my memoir:

"Asciidoc was not really designed for something the size of the Exim manual. Not only was it quite slow, being written in Python, but it had difficulty coping with all the typographic variations needed in something the size of a book."

The Exim manual was (is still, I hope) the size of a book: I offered it as a PDF, as plain text (for easy searching), and as HTML. Asciidoc created DocBook XML which could be processed into all these formats. In the end I wrote my own front-end processor (xfpt). PCRE was different. At the start it didn't seem to need a lot of documentation, so I just went for a man page, which later expanded into a number of man pages (and then web pages).

I expect Asciidoc is very different to what it was 20 years ago, and none of the PCRE2 pages are anywhere near the length of a book, with fewer typographic requirements, so if it works for you...  It is definitely true that the man/roff format is rough and outdated.

Regards,

Philip

[Nicholas Wilson]

Thank you very much Philip,

I am glad to appreciate PCRE2's long history, and I'm just glad to have a little time to pause and tidy things, before another release of code changes I have planned.

Those pieces of PCRE2 history are valuable, thank you.

All the best,

Nick

[Matthew Vernon]

Hi,

These changes all seem reasonable to me :)

I know everyone has opinions on C layout, but I think as long as the
format is reasonably clear (and documented), you get to pick what works
for you.

I've no real exposure to Asciidoc, but if you think markdown isn't up to
the task, it seems a sensible way to go.

Regards,

Matthew

[Philip Hazel]

Out of interest, I have just dug out the documentation for PCRE release 1.00 (1997). It consists of a single man page, 1017 lines long, with a separate man page for what was then called pgrep. The test data consists of two files - testinput and testoutput. How much has changed! 

Regards,

Philip

--
You received this message because you are subscribed to the Google Groups "PCRE2 discussion list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pcre2-dev+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/pcre2-dev/784dfae3-3f66-4fff-b6ae-097a416ccfc0n%40googlegroups.com.

[Matthew Vernon]

Hi,

One further thought, and apologies if this is teaching you to suck eggs,
but pandoc should be able to do roff to asciidoc conversion for you,
which might at least give you a decent starting point.

Regards,

Matthew

[Nicholas Wilson]

Hi Matthew,

Not at all, that's a very helpful suggestion. I hadn't given any thought yet to _how_ I would convert all the docs to Asciidoc, so pandoc could well the right tool.

My main reason for wanting to use Asciidoc over Markdown is simply that it seems to generate much better manpage output. Markdown is not a very rich format, and while there are plenty of Markdown tools for everything under the sun, it's certainly not designed for processing into Unix man.

Status of these three tasks

1. The addition of RunTest.py and RunGrepTest.py, and removal of Run(Grep)Test.bat, is complete.

If there are any problems, I would be happy to fix them. In particular, MinGW, Cygwin, and various other in-between environments seem to be challenging to get right. I do have CI/CD testing for those, but it's not perfect.

2. Code reformatting is done! You may love it or hate it, so if there's anything you (as contributors) think I ought to change, there is a chance. After a little time has elapsed, I will probably refuse to reformat again (churn is a waste of energy), but at the moment, I'm still open to requests.

The indentation is quite standard, and not too different from what we had before (braces on a fresh line by themselves).

I used clang-format in the end to make the code almost-entirely regular. I had been wondering about whether that was worth it, but I decided that the easiest way to accept contributions is simply to have a definite and simple set of rules.

There is no CI reformatting. No enforcement - no change to anyone's workflows.

I have not made braces mandatory: you can still do:

if (xyz)

    then_sleep();

However I have added line-breaks throughout, so that then_sleep() is never on the same line as the if(). I find it (a little) preferable not to hide control flow halfway along a line, and I like the ability to scan down the ragged left edge with my eye.

I have made braces uniform on if/else chains, so that if one branch is braced, the others are too.

Finally, I have added newlines after every terminal if, else, and do/while. This makes it clear which blocks are connected, and avoids confusing a new if-else-chain for a continuation of the previous.

if (xyz)

{ ... }

        // <- blank line added here.

if (abc)

{ ...}

In conclusion: it's moderately lenient. Many corporations now require all if/else conditions to have braces. I've added a few, judiciously, but not made the code excessively sprawling.

3. The Asciidoc change is probably on pause. I'll get to it some other month...

There's more useful bugfixing to do on PCRE2!

All the best,

Nick

[Philip Hazel]

If you do use Asciidoc and it makes DocBook XML, as long as it is the subset known as Simple DocBook, my Simple DocBook Processor (https://github.com/PhilipHazel/SDoP) can turn it into PostScript, which can then be made into a PDF. However, the PCRE documentation has never been offered in PDF format, and there are no doubt also other ways to do the same thing, but I just thought I'd mention it.