new projects/external/mksh proposal… or replace sh?

[mirabilos]

Hi,

I’m the maintainer of the MirBSD Korn Shell – http://mirbsd.de/mksh –
and have successfully
built a full Android tree with mksh (in two flavours, ① full+printf, ②
small) as an external project
(I’d like to not assign over copyright or equivalent for this, but
still like to get it integrated).

To the developer, it has tab completion, adb can be built withOUT the
evil -DSH_HISTORY
that causes so many problems, and it can be used as a minimal
scripting language as well.

The licence is similar to BSD, DFSG- and FSF-free and OSI and OKFN
approved. Some
auxiliary code (that can be omitted – setmode.c for the optional mknod
(1) builtin, and
printf.c for the optional printf(1) builtin) are under the 3-clause
UCB BSD licence.

If someone thinks this would benefit a replacement of the crappy
NetBSD® ash in its
ancient version, I’d love to hack it into projects/system/core/mksh
and retire sh. On the
bad side, it’s a little larger – on the bright side, it has Korn shell
functions (such as safe
[[ comparision), bash/ksh93/zsh-like extensions, Emacs (and, optional)
Vi editing modes,
and isn’t MUCH larger. It’s also used as /bin/sh by FreeWRT Embedded
GNU/Linux ADK,
so the precedence is set, and it’s very portable and embed-friendly.

-rwxr-xr-x 1 2423 5010 168232 Dec 1 16:56 out/target/product/generic/
system/bin/mksh
-rwxr-xr-x 1 2423 5010 139104 Dec 1 16:57 out/target/product/generic/
system/bin/mksh-small
-rwxr-xr-x 1 2423 5010 86848 Dec 1 15:37 out/target/product/generic/
system/bin/sh

Requirement is the main tree plus Gerrit changes 12676 and 12677.

These have been built from projects/external/mksh. I’ve also
successfully created an
NDK application tree from it, targetting Android 1.5 (with sys_signame
[] provided by
the build instead of Bionic libc).

Now, how do we go on? Is there interest? Should I publish what I
already have?
Where?

By the way, since I couldn’t get the emulator to do so, which encoding
is used by default?
When I print 'm\xe4h' or 'm\ue4h' I get 'mh' both times, seems as if
Terminal were 7bit?
mksh by default looks at setlocale and nl_langinfo(CODESET), which is
not available, so
the options are: look at ${LC_ALL:-${LC_CTYPE:-${LANG:-C}}}, enable
UTF-8 mode by
default, disable UTF-8 mode by default. The last two reduce code size
a little (not much);
I’ve chosen the latter option for now, but, since adb is usually used
from UTF-8 environments,
would enable UTF-8 by default if Terminal is indeed 7bit.

Thanks in advance,
//mirabilos
--
I believe no one can invent an algorithm. One just happens to hit upon
it
when God enlightens him. Or only God invents algorithms, we merely
copy them.
If you don't believe in God, just consider God as Nature if you won't
deny
existence. -- Coywolf Qi Hunt

[mirabilos]

Hi again,

no feedback at all? Does nobody desire a shell with proper history,
tab completion, more secure builtins, etc. at not even two times
the size, either as /bin/sh or at least for development purposes?

[Joshua D]

I'm currently using Bourne-again shell (BASH), but not sure if others
keep default.
Would be good to see a decent shell running on Android, the current
one is lacking.

I guess it's just like busybox (everyone uses it, except the AOSP) -
But perhaps due to licensing? I'm not sure.

[Ryan Beesley]

I'm more familiar with BASH, but I think any improvement to the current SH would be welcome.

/Ryan

--

You received this message because you are subscribed to the Google Groups "android-platform" group.
To post to this group, send email to android-...@googlegroups.com.
To unsubscribe from this group, send email to android-platfo...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/android-platform?hl=en.

[Dennis.Yxun]

good to hear~
I would like a more advanced shell, I like tab completion
current one is quite dumb and lack functionality

for the binary size, not that much big, it should be acceptable

--

[Joshua D]

This to me seems a tough one. the shell the op mentions sounds good,
what does the ops suggestion have that bash dosnt? I think any shell
would be an improvement, but are we later going to want to move to
bash?
JD

On Dec 18, 12:22 pm, "Dennis.Yxun" <dennis.y...@gmail.com> wrote:
> good to hear~
> I would like a more advanced shell, I like tab completion
> current one is quite dumb and lack functionality
>
> for the binary size, not that much big, it should be acceptable
>
> On Wed, Dec 16, 2009 at 9:46 PM, mirabilos

> <tg+m...@mirbsd.org<tg%2Bm...@mirbsd.org>

>
>
>
> > wrote:
> > Hi again,
>
> > no feedback at all? Does nobody desire a shell with proper history,
> > tab completion, more secure builtins, etc. at not even two times
> > the size, either as /bin/sh or at least for development purposes?
>
> > --
>
> > You received this message because you are subscribed to the Google Groups
> > "android-platform" group.
> > To post to this group, send email to android-...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > android-platfo...@googlegroups.com<android-platform%2Bunsubscrib e...@googlegroups.com>

[mirabilos]

For one thing, licencing. GNU bash is, well, GNU stuff; mksh is BSD
stuff.
Other than that, well, compare. For example, i386 linux glibc:

-rwxr-xr-x 1 root root 702160 May 12 2008 /bin/bash*
-rwxr-xr-x 1 root root 203132 Nov 23 14:10 /bin/mksh*

And that’s dynamically linked (mksh: libc, libbsd; bash: libc, libdl,
libncurses!).

Third, it is mostly bash/ksh93/zsh compatible and gets an increasing
featureset, but is a Korn shell and as such has some unique (except
ksh93 has them foo) functionality like co-processes, but for Android,
the thing I think most of you would find appealing is that it offers
good interactive behaviour (history, tab completion, etc.) that just
works (no ~/.inputrc, no ncurses, no readline) in a very small binary.

Here’s one (binary) for you to play with, to see how it “feels” like:
http://www.mirbsd.org/MirOS/dist/mir/mksh/ports/mksh-39.9.20091128-android15.tgz
Manpage: http://www.mirbsd.org/MirOS/dist/mir/mksh/mksh.pdf

Besides, GNU bash is so much a GNU/Linux thing, while Android
takes more of the tools from ancient NetBSD® so this fits just right
in. Korn Shell is the default on very many Unicēs.

[Dennis.Yxun]

mirabilos:
    Great job!
    Would you kind public the source?
 I have a MIPS platform and would like to give a try

Dennis
 

--

You received this message because you are subscribed to the Google Groups "android-platform" group.
To post to this group, send email to android-...@googlegroups.com.

To unsubscribe from this group, send email to android-platfo...@googlegroups.com.

[mirabilos]

On Dec 22, 10:20 am, "Dennis.Yxun" <dennis.y...@gmail.com> wrote:

>     Would you kind public the source?

Sure: https://eurynome.mirbsd.org/~tg/pub/android-ndk-1.6_r1~apps~mksh.tgz
is what I (think I) used to build that binary. Note it’s a draft, and
integration as
either projects/external/mksh or replacing /bin/sh will be done (only
slightly)
differently.

I just want to warm you people up to the idea first.

[Dennis.Yxun]

thanks , I may try it later and keep posting the result here

[Dennis.Yxun]

HI Thorsten:
     I've integrated into my android platform, just simply throw into the system
compile and run, then it works, perfect~
     Btw, I'm testing under MIPS platform
     Thank you for your great job

Dennis

[shaunak saha]

Hi Mirabilos,

Thanks.Great job!

I just tried it.Its working fine for me.Tab completion is very helpful.

 

Thanks,

Shaunak

[mirabilos]

So,

now that people like it… what about my original question, whom do I
contact to get that thing integrated?

[David Turner]

You submit it to r.android.com as all other enhancements. This requires you signing the Contributor License Agreement.

Then your patches will be reviewed and discussed by OHA engineers.

[mirabilos]

Two things…

One, it’s not possible since I cannot “checkout” a nonexisting
repository, it would
need to be created first.

(Or I’ll just send a patch that replaces the current sh… that’d be
even nicer.)

Second, will the licence terms (BSD/MIT-style) on the original source
code that
would be imported be preserved?

[David Turner]

On Fri, Jan 8, 2010 at 1:43 AM, mirabilos <tg+...@mirbsd.org> wrote:

Two things…

One, it’s not possible since I cannot “checkout” a nonexisting
repository, it would
need to be created first.

(Or I’ll just send a patch that replaces the current sh… that’d be
even nicer.)

Ah, good point. Replacing the current sh seems to be the simplest way for now.

 

Second, will the licence terms (BSD/MIT-style) on the original source
code that
would be imported be preserved?

Yes, as long as the license is BSD/MIT/Apache, it will be accepted.

[Chih-Wei]

Nice work!
Once you provide a patch (either in external/mksh or system/core)
I would like to try it on android-x86.

--
Android-x86 project
http://www.android-x86.org

[Dennis.Yxun]

Hi Chih-Wei:
     I've already tried on android-mips, it works.
    So I think it should be quite straight forward for you to try under x86.
One thing, if you want to try to call mksh via adb, you probably should modify adb source code
change SHELL_COMMAND to /system/bin/mksh or simply overwrite /system/bin/sh with mksh

[mirabilos]

Ok,, I’ve done so now.

You first need to apply this change:
https://review.source.android.com/#change,12676 • Change Iae258589:
Add declaration for sys_signame[] forgotten in change 12489

I have no idea how to mark it as dependency in Gerrit.

Then you apply the series of changes amounting to this:
https://review.source.android.com/#change,13148 • Change I5b5c9e78:
shlist.c is no longer needed without -DSH_HISTORY

That’s my replace-sh-with-mksh branch.

I’ve done a “make”, created my own AVD from it and booted it
successfully.
The shell works fine. The only thing missing is a default /.mkshrc
file, for
example for a nice prompt, but then, it’s already WAY better than the
old
NetBSD® ash-based shell.

Please test!

Thanks in advance,
//mirabilos

[mirabilos]

Upgraded to mksh R39b, released today.
No really major changes, mostly minor bugfixes;
the snapshot was really close to the release anyway.

Now I think it’s ok for inclusion. I made an AVD out
of this and tested it in the emulator.

[mirabilos]

FYI:

Chih-Wei Huang just mailed round that mksh has been merged
into eclair-x86 and that it is repo-sync-able.

Thanks for the acceptance!

I’ll continue to provide updates via Gerrit, if that is desired.

[mirabilos]

Ok, updated again, and rebased onto a freshly checked out repo.
(Please review at Gerrit again, the old 「+1」 was lost due to the
rebase process.)

You need the following change in bionic, as usual:
https://review.source.android.com/12676
repo download platform/bionic 12676/2

Then you get system/core to mksh with these changes (series):
(up to) https://review.source.android.com/14102
repo download platform/system/core 14102/1

New in this version:
• more up to date mksh base code
• EXECSHELL is now /system/bin/sh by default, fixing “adb install”
and other things; thanks to Chih-Wei Huang
• there is a /system/etc/mkshrc which sets up a few environment
variables, shell aliases, and a default prompt

XXX $HOME defaults to /data but now that I boot it I can see that the
user
XXX has no permissions on that; better suggestion other than "/"
anyone?

The EXECSHELL thing was responsible for failing to run the command
/system/bin/pm which does not begin with a shebang line (the shell
falls back to using ${EXECSHELL:-/bin/sh} then); I’ve made the default
configurable at compile-time in the upstream code, same for the path
to ~/.mkshrc (which are strictly(!) intended to be used for single-
user
and/or embedded systems, not for regular distro packages, mind you).

I also disabled the mknod(8) builtin, so that setmode.c is also no
longer compiled (licence and space reasons; the old sh had none,
and people probably don’t need it – otherwise just tell me and I
will re-enable it).

I kept the printf builtin in. It can be kept, gotten rid of, or even
(with
some glue code I need to hack, less than a dozen lines probably)
be made a toolbox-like approach with a printf→mksh symlink so
that it’s exposed to other apps, if you like. I recommend, however,
to keep printf’s floating point code (the rest of mksh has none, but
printf is third-party code from BSD) disabled…

If the shell’s too big, I suggest disabling the Vi editing mode first
by using -DMKSH_S_NOVI when compiling. If the result would still
be too big, talk to me. I’d like to get mksh into Android, and ex-
pose its users to as much of its useful functionality as possible.
The current size, which I think is reasonable, is:
-rwxr-xr-x 1 tglase 151888 Mar 28 00:02 out/target/product/generic/
system/bin/sh*

The code (including the EXECSHELL change, but not the mkshrc)
has undergone testing via Android-x86, so I think it’s (almost… see
the $HOME issue above) ready for mainstream, if you want it.

By the way, does anyone know how I can make the changes from
system/core depend on the change from bionic? Gerrit dependen-
cies appear to be within the same git part-repository, not project-
wide…

Thanks for listening!
//mirabilos
--
FWIW, I'm quite impressed with mksh interactively. I thought it was
much
*much* more bare bones. But it turns out it beats the living hell out
of
ksh93 in that respect. I'd even consider it for my daily use if I
hadn't
wasted half my life on my zsh setup. :-) -- Frank Terbeck in #!/bin/
mksh

[mirabilos]

I forgot to note:

I have disabled -fstack-protector-all in the explicit flags,
both for the NDK (flag test) and final build. On Android,
in contrast to Android-x86, it is still built with ProPolice
(-fstack-protector) on because that’s a global flag, so
no need to worry there.

I also nuked the NDK (app build) and mksh-small parts
from Android.mk – if anybody had use for them nowadays,
they’re easy to get back I think, but they were not worth
the additional maintenance, and I wanted it cleaned up
for possible inclusion.

//m