[ANN] ksh 93u+m 1.0.0-beta.1

79 views
Skip to first unread message

Martijn Dekker

unread,
May 10, 2021, 12:57:48 PM5/10/21
to Korn Shell
Announcing: KornShell 93u+m 1.0.0-beta.1
https://github.com/ksh93/ksh

In May 2020, when every KornShell (ksh93) development project was
abandoned, development was rebooted in a new fork based on the last
stable AT&T version: ksh 93u+. Now, one year and hundreds of bug fixes
later, the first beta version is ready, and KornShell lives again. This
new fork is called ksh 93u+m as a permanent nod to its origin; a standard
semantic version number is added starting at 1.0.0-beta.1. Please test
the beta and report any bugs you find, or help us fix known bugs.

Main developers: Martijn Dekker, Johnothan King, hyenias

Contributors: Andy Fiddaman, Anuradha Weeraman, Chase, Gordon Woodhull,
Govind Kamat, Harald van Dijk, Lev Kujawski, Marc Wilson, Sterling Jensen

HOW TO GET IT

Please download the source code tarball from our GitHub releases page:

https://github.com/ksh93/ksh/releases

To build, follow the instructions in README.md or src/cmd/ksh93/README.
Hopefully, OS/distro packagers will make ksh 93u+m packages available soon.
If you would like to have a binary for your OS from us, ask and we'll try
to build one and add it to the releases page.

HOW TO GET INVOLVED

To report a bug, please open an issue at our GitHub page (see above).
Alternatively, email me at mar...@inlv.org with your report.
To get involved in development, read the brief policy information in
README.md and then jump right in with a pull request or email a patch.
See the TODO file in the top-level directory for a to-do list.

MAIN CHANGES SINCE KSH 93U+ 2012-08-01

Hundreds of bugs have been fixed, including many serious/critical bugs.
This includes upstreamed patches from OpenSUSE, Red Hat, and Solaris, fixes
backported from the abandoned 93v- beta and ksh2020 fork, as well as many
new fixes from the community. See the NEWS file for more information, and
the git commit log for complete documentation of every fix. Incompatible
changes have been minimised, but not at the expense of fixing bugs. For a
list of potentially incompatible changes, see src/cmd/ksh93/COMPATIBILITY.

Though there was a "no new features, bugfixes only" policy, some new
features were found necessary, either to fix serious design flaws or to
complete functionality that was evidently intended, but not finished.
Below is a summary of these new features.

New command line editor features:

- The forward-delete and End keys are now handled as expected in the
emacs and vi built-in line editors.

- In the vi and emacs line editors, repeat count parameters can now also
be used for the arrow keys and the forward-delete key. E.g., in emacs
mode, <ESC> 7 <left-arrow> will now move the cursor seven positions to
the left. In vi control mode, this would be entered as: 7 <left-arrow>.

New shell language features:

- The &>file redirection shorthand (for >file 2>&1) is now available for
all scripts and interactive sessions and not only for profile/login
scripts, bringing ksh 93u+m in line with mksh, bash, and zsh.

- File name generation (a.k.a. pathname expansion, a.k.a. globbing) now
never matches the special navigational names '.' (current directory)
and '..' (parent directory). This change makes a pattern like .*
useful; it now matches all hidden files (dotfiles) in the current
directory, without the harmful inclusion of '.' and '..'.

- Tilde expansion can now be extended or modified by defining a
.sh.tilde.get or .sh.tilde.set discipline function. This replaces a
2004 undocumented attempt to add this functionality via a .sh.tilde
command, which never worked and crashed the shell. See the manual for
details on the new method.

New features in built-in commands:

- Usage error messages now show the --help/--man self-documentation options.

- Path-bound built-ins (such as /opt/ast/bin/cat) can now be executed by
invoking the canonical path, so the following will now work as expected:
$ /opt/ast/bin/cat --version
version cat (AT&T Research) 2012-05-31

- 'command -x' now looks for external commands only, skipping built-ins.
In addition, its xargs-like functionality no longer freezes the shell on
Linux and macOS, making it effectively a new feature on these systems.

- 'redirect' now checks if all arguments are valid redirections before
performing them. If an error occurs, it issues an error message instead
of terminating the shell.

- 'suspend' now refuses to suspend a login shell, as there is probably no
parent shell to return to and the login session would freeze.

- 'times' now gives high precision output in a POSIX compliant format.

- 'typeset' now gives an informative error message if an incompatible
combination of options is given.

- 'whence -v/-a' now reports the location of autoloadable functions.

New features in shell options:

- A new --globcasedetect shell option is added on OSs where we can
check for a case-insensitive file system (currently Windows/Cygwin,
macOS, Linux and QNX 7.0+). When this option is turned on, file name
generation (globbing), as well as file name tab completion on
interactive shells, automatically become case-insensitive on file
systems where the difference between upper and lower case is ignored
for file names. This is transparently determined for each directory, so
a path pattern that spans multiple file systems can be part
case-sensitive and part case-insensitive.

- A new --nobackslashctrl shell option disables the special escaping
behaviour of the backslash character in the emacs and vi built-in
editors. Particularly in the emacs editor, this makes it much easier to
go backward, insert a forgotten backslash into a command, and then
continue editing without having your next cursor key replace your
backslash with garbage. Note that Ctrl+V (or whatever other character
was set using 'stty lnext') always escapes all control characters in
either editing mode.

- A new --posix shell option has been added to ksh 93u+m that makes the
ksh language more compatible with other shells by following the POSIX
standard more closely. See the manual page for details. It is enabled by
default if ksh is invoked as sh, otherwise it is disabled by default.

- Enhancement to -G/--globstar: symbolic links to directories are now
followed if they match a normal (non-**) glob pattern. For example, if
'/lnk' is a symlink to a directory, '/lnk/**' and '/l?k/**' now work as
you would expect.

sambea...@gmail.com

unread,
May 10, 2021, 3:06:58 PM5/10/21
to Korn Shell
Is anyone else having trouble building this version? It seems to be missing all of the Makefiles...

Marc Wilson

unread,
May 10, 2021, 3:25:28 PM5/10/21
to Korn Shell
Are you following the build procedure laid out in the README?

## Build   

To build ksh with a custom configuration of features, edit

[`src/cmd/ksh93/SHOPT.sh`](https://github.com/ksh93/ksh/blob/master/src/cmd/ksh93/SHOPT.sh).   

Then `cd` to the top directory and run:

```sh

bin/package make

```

ksh93u+m uses the AST build system as ksh always has.


Sam Alleman

unread,
May 10, 2021, 3:43:18 PM5/10/21
to Marc Wilson, Korn Shell
Yes. I even downloaded ast-master and verified that I was able to build that package before I posted here. Where ast-master builds successfully, 
I get the following error message from ksh93u+m:

package: update /tmp/ksh-1.0.0-beta.1/bin/execrate
package: update /tmp/ksh-1.0.0-beta.1/bin/ignore
package: update /tmp/ksh-1.0.0-beta.1/bin/mamprobe
package: update /tmp/ksh-1.0.0-beta.1/bin/silent
package: warning: yacc: not found -- some make actions may fail
package: warning: bison: not found -- some make actions may fail
package: update /tmp/ksh-1.0.0-beta.1/arch/linux.i386-64/bin/ar
package: update /tmp/ksh-1.0.0-beta.1/arch/linux.i386-64/lib/probe/C/make/probe
package: update /tmp/ksh-1.0.0-beta.1/arch/linux.i386-64/bin/mamake
package: update /tmp/ksh-1.0.0-beta.1/arch/linux.i386-64/bin/proto
package: update /tmp/ksh-1.0.0-beta.1/arch/linux.i386-64/bin/ratz
package: update /tmp/ksh-1.0.0-beta.1/arch/linux.i386-64/bin/release
package: make output captured in /tmp/ksh-1.0.0-beta.1/arch/linux.i386-64/lib/package/gen/make.out
package: make start at Mon May 10 14:36:24 CDT 2021 in /tmp/ksh-1.0.0-beta.1/arch/linux.i386-64
CC=cc
SHELL=/usr/bin/ksh
HOSTTYPE=linux.i386-64
NPROC=16
PACKAGEROOT=/tmp/ksh-1.0.0-beta.1
INSTALLROOT=/tmp/ksh-1.0.0-beta.1/arch/linux.i386-64
PATH=/tmp/ksh-1.0.0-beta.1/arch/linux.i386-64/bin:/tmp/ksh-1.0.0-beta.1/bin:/usr/bin:/usr/sbin:/sbin:/bin:/usr/local/bin:/usr/local/sbin:/home/sja/bin:/home/sja/SU/bin:/usr/games:/usr/local/games:.:/c/windows:/c/windows/system32
make: a makefile must be specified when Nmakefile,nmakefile,Makefile,makefile omitted
package: make: errors making /tmp/ksh-1.0.0-beta.1/arch/linux.i386-64/bin/ksh
package: make done  at Mon May 10 14:36:24 CDT 2021 in /tmp/ksh-1.0.0-beta.1/arch/linux.i386-64


Sam Alleman
LSP Instructor #484
(337) 205-3216


--
You received this message because you are subscribed to a topic in the Google Groups "Korn Shell" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/korn-shell/2VK8kS0_VhA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to korn-shell+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/korn-shell/d762eab8-4e80-40b9-9204-4070b1662a2en%40googlegroups.com.

Martijn Dekker

unread,
May 10, 2021, 4:43:47 PM5/10/21
to korn-...@googlegroups.com
Op 10-05-21 om 20:40 schreef Sam Alleman:
> Yes. I even downloaded ast-master and verified that I was able to build
> that package before I posted here. Where ast-master builds successfully,
> I get the following error message from ksh93u+m:

Hmmm. We removed nmake in ksh 93u+m because it had too many problems to
keep using and we simply didn't have the resources/motivation to fix it.

Do you have a version of nmake in your $PATH somewhere? The build system
probably still has code that finds and tries to use it. Which won't
work, because, as you've observed, all the Makefiles are gone.

Will the build work if you temporarily remove nmake in your $PATH?

If so, that's the issue, and I will have to track down that code and
remove it from the build system.

- Martijn

--
|| modernish -- harness the shell
|| https://github.com/modernish/modernish
||
|| KornShell lives!
|| https://github.com/ksh93/ksh

Sam Alleman

unread,
May 10, 2021, 5:18:47 PM5/10/21
to Martijn Dekker, korn-shell
Yes, I had nmake in my path and that was the issue. I excluded /usr/local/bin from my PATH and was able to build ksh 93u+m. Thank you so much! :) 

Martijn Dekker

unread,
May 10, 2021, 5:32:07 PM5/10/21
to korn-...@googlegroups.com
Op 10-05-21 om 21:43 schreef Martijn Dekker:
> Will the build work if you temporarily remove nmake in your $PATH?
>
> If so, that's the issue, and I will have to track down that code and
> remove it from the build system.

I've confirmed that this is the case by building nmake from the original
AT&T distribution and installing it in my $PATH.

Please find attached a minimal patch that disables the detection of
nmake in the build system. Apply from the top-level directory using:

patch -p1 <ksh-nonmake.patch

After that, everything should build. Please let me know if you still
encounter problems.
ksh-nonmake.patch

Sam Alleman

unread,
May 10, 2021, 5:37:59 PM5/10/21
to Martijn Dekker, korn-shell
I reverted to my normal path, applied the patch you sent, and it is now happily building. That fixed my issue. Thanks. 

Sam Alleman
LSP Instructor #484
(337) 205-3216

--
You received this message because you are subscribed to the Google Groups "Korn Shell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to korn-shell+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/korn-shell/cb560ae7-5e5b-2736-2623-810c06798c90%40inlv.org.

Kamil Dudka

unread,
May 11, 2021, 3:27:58 AM5/11/21
to Korn Shell, Martijn Dekker
On Monday, May 10, 2021 6:57:44 PM CEST Martijn Dekker wrote:
> Announcing: KornShell 93u+m 1.0.0-beta.1
> https://github.com/ksh93/ksh
>
> In May 2020, when every KornShell (ksh93) development project was
> abandoned, development was rebooted in a new fork based on the last
> stable AT&T version: ksh 93u+. Now, one year and hundreds of bug fixes
> later, the first beta version is ready, and KornShell lives again. This
> new fork is called ksh 93u+m as a permanent nod to its origin; a standard
> semantic version number is added starting at 1.0.0-beta.1. Please test
> the beta and report any bugs you find, or help us fix known bugs.
>
> Main developers: Martijn Dekker, Johnothan King, hyenias
>
> Contributors: Andy Fiddaman, Anuradha Weeraman, Chase, Gordon Woodhull,
> Govind Kamat, Harald van Dijk, Lev Kujawski, Marc Wilson, Sterling Jensen

Good job. Thanks to all developers and contributors! I am able to build it
on Fedora rawhide. Two regression tests (locale and pty) are failing in my
environment. Test log is attached.

Kamil
f35-ksh-shtests.log.gz

Martijn Dekker

unread,
May 11, 2021, 4:37:20 PM5/11/21
to Kamil Dudka, Korn Shell
Op 11-05-21 om 08:27 schreef Kamil Dudka:
> Good job. Thanks to all developers and contributors! I am able to build it
> on Fedora rawhide. Two regression tests (locale and pty) are failing in my
> environment. Test log is attached.

Thanks for the report.

This failure is easy to fix:

locale.sh[84]: 'read' doesn't skip multibyte input
correctly (ja_JP.ujis, \x95\x5c)

This is a problem with the test; this Shift-JIS specific test should not
be run in a non-Shift-JIS locale. So the fix is to skip it unless the
locale string ends in '.SJIS' (case insensitive). This is now committed
to git and will be in the next beta.

The pty failures are harder. (pty is a pseudoterminal program used to
test the interactive shell.) I was able to get it to fail when running
Fedora-Workstation-Live-x86_64-34-1.2 as a live DVD in a VirtualBox VM.
It fails differently on every run. So it seems like pty has some race
condition on Fedora. Which is odd, because seems to work fine on every
other Linux distro. Even odder is that, after installing Fedora onto the
VM's "hard disk" and booting it regularly, I can no longer reproduce the
pty test failures. Is there anything your team can do to debug this?

- Martijn

Kamil Dudka

unread,
May 12, 2021, 5:34:46 AM5/12/21
to Martijn Dekker, Korn Shell, Vincent Mihalkovic
On Tuesday, May 11, 2021 10:37:15 PM CEST Martijn Dekker wrote:
> Op 11-05-21 om 08:27 schreef Kamil Dudka:
> > Good job. Thanks to all developers and contributors! I am able to build
> > it on Fedora rawhide. Two regression tests (locale and pty) are failing
> > in my environment. Test log is attached.
>
> Thanks for the report.
>
> This failure is easy to fix:
>
> locale.sh[84]: 'read' doesn't skip multibyte input
> correctly (ja_JP.ujis, \x95\x5c)
>
> This is a problem with the test; this Shift-JIS specific test should not
> be run in a non-Shift-JIS locale. So the fix is to skip it unless the
> locale string ends in '.SJIS' (case insensitive). This is now committed
> to git and will be in the next beta.

Thanks for the quick fix!

I cannot see the failure with upstream v1.0.0-beta.1-6-g8a953e8 any more.

> The pty failures are harder. (pty is a pseudoterminal program used to
> test the interactive shell.) I was able to get it to fail when running
> Fedora-Workstation-Live-x86_64-34-1.2 as a live DVD in a VirtualBox VM.
> It fails differently on every run. So it seems like pty has some race
> condition on Fedora. Which is odd, because seems to work fine on every
> other Linux distro. Even odder is that, after installing Fedora onto the
> VM's "hard disk" and booting it regularly, I can no longer reproduce the
> pty test failures. Is there anything your team can do to debug this?

Yes, it seems to be a timing issue. I was running it in kvm-based VM.
The failure goes away if I enable valgrind:

$ bin/shtests pty
#### Regression-testing /home/kdudka/ksh/arch/linux.i386-64/bin/ksh ####
test pty begins at 2021-05-12+11:18:12
pty.sh[625]: nobackslashctrl in emacs: line 634: expected "^:test-2: \r\n$", got "set -o emacs --nobackslashctrl\r\n"
test pty failed at 2021-05-12+11:18:28 with exit code 1 [ 40 tests 1 error ]
test pty(C.UTF-8) begins at 2021-05-12+11:18:28
test pty(C.UTF-8) passed at 2021-05-12+11:18:44 [ 40 tests 0 errors ]
test pty(shcomp) begins at 2021-05-12+11:18:44
shcomp-pty.ksh[673]: emacs backslash escaping: line 682: expected "^:test-2: set -o emacs$", got ":test-1: set -o emacs\r\n"
test pty(shcomp) failed at 2021-05-12+11:18:58 with exit code 1 [ 40 tests 1 error ]
Total errors: 2
CPU time user: system:
main: 0m00.004s 0m00.025s
tests: 0m00.456s 0m04.088s

$ bin/shtests --valgrind pty
#### Regression-testing /home/kdudka/ksh/arch/linux.i386-64/bin/ksh ####
test pty begins at 2021-05-12+11:26:06
test pty passed at 2021-05-12+11:26:26 [ 40 tests 0 errors ]
test pty(C.UTF-8) begins at 2021-05-12+11:26:26
test pty(C.UTF-8) passed at 2021-05-12+11:26:46 [ 40 tests 0 errors ]
test pty(shcomp) begins at 2021-05-12+11:26:46
test pty(shcomp) passed at 2021-05-12+11:27:05 [ 40 tests 0 errors ]
Total errors: 0
CPU time user: system:
main: 0m00.008s 0m00.035s
tests: 0m06.545s 0m08.633s

I was thinking about disabling stdio buffers but the program seems to use
sfio instead, which uses its own buffering I guess.

Kamil

> - Martijn


Reply all
Reply to author
Forward
0 new messages