Autogen in Emacs Shell

[Alexander Shukaev]

Hello,

I am considering to give up my current terminal (ConEmu) for Emacs Shell at
least for some simple tasks. I'm on Windows and use MSYS2 as a primary
development environment. I just wanted to build Emacs from Emacs Shell and
here is what I've experienced when running `autogen.sh':

Checking for autoconf (need at least version 2.65)...
> ok
> Checking for automake (need at least version 1.11)...
> missing
> Building Emacs from the repository requires the following specialized
> programs:
> autoconf (minimum version 2.65)
> automake (minimum version 1.11)
> Your system seems to be missing the following tool(s):
> automake (missing)
> If you think you have the required tools, please add them to your PATH
> and re-run this script.
> Otherwise, please try installing them.
> On systems using rpm and yum, try: "yum install PACKAGE"
> On systems using dpkg and apt, try: "apt-get install PACKAGE"
> Then re-run this script.
> If you do not have permission to do this, or if the version provided
> by your system is too old, it is normally straightforward to build
> these packages from source. You can find the sources at:
> ftp://ftp.gnu.org/gnu/PACKAGE/
> Download the package (make sure you get at least the minimum version
> listed above), extract it using tar, then run configure, make,
> make install. Add the installation directory to your PATH and re-run
> this script.
> If you know that the required versions are in your PATH, but this
> script has made an error, then you can simply run
> autoreconf -i -I m4
> instead of this script.
> Please report any problems with this script to bug-gn...@gnu.org .


That's weird because immediately issuing

which automake


says

/usr/bin/automake


I use `bash' as Emacs Shell. In ConEmu the same `autogen.sh' works fine.

Furthermore, when using `bsdtar' from Emacs Shell, I saw:

bsdtar: Failed to set default locale


What could be the problem here?

Regards,
Alexander

[to...@tuxteam.de]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, Apr 29, 2015 at 04:36:04AM +0200, Alexander Shukaev wrote:
> Hello,
>
> I am considering to give up my current terminal (ConEmu) for Emacs Shell at

> least for some simple tasks [...]

> That's weird because immediately issuing
>
> which automake
>
>
> says
>
> /usr/bin/automake
>
>
> I use `bash' as Emacs Shell. In ConEmu the same `autogen.sh' works fine.
>
> Furthermore, when using `bsdtar' from Emacs Shell, I saw:
>
> bsdtar: Failed to set default locale
>
>
> What could be the problem here?

As a hunch (but still, there are things this hunch can't explain), perhaps
ConEmu is running your bash as a "login shell", meaning that bash does some
environment settings it won't redo in a "normal" shell session, which assumes
it inherits those from its ancestor login shell.

An easy way to verify this hunch would be to tell your (Emacs) terminal
to use "/bin/bash --login" instead of just "/bin/bash" as thedefault shell
(please substitute /bin/bash by whatever the correct path is in your system).

Another thing you could do is to issue in both shells the built-in command
"set" (which shows you all the environment variable settings) and study
the differences. Watch out especially for PATH, LANG, LANGUAGE and all the
other locale stuff, like LC_*

hth
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlVAltYACgkQBcgs9XrR2kZr3gCfcoHZD2m72tyUs8i0WPt8Q9Fw
2iEAniYIhNQMYOSLft1ffAcnuS6oxqCM
=TlzF
-----END PGP SIGNATURE-----

[Eli Zaretskii]

> Date: Wed, 29 Apr 2015 04:36:04 +0200
> From: Alexander Shukaev <haro...@gmail.com>

>
> I am considering to give up my current terminal (ConEmu) for Emacs Shell at

> least for some simple tasks.

You mean, Eshell, as in "M-x eshell RET"? Or something else?

> I'm on Windows and use MSYS2 as a primary
> development environment. I just wanted to build Emacs from Emacs Shell and
> here is what I've experienced when running `autogen.sh':
>
> Checking for autoconf (need at least version 2.65)...

How do you even run autogen.sh from Eshell on Windows? I get

./autogen.sh: Permission denied

and after I force Eshell to realize that Windows has no execute bit
(by setting eshell-force-execution non-nil), I get this:

/bin/sh: command not found

So if you got past that, you have a lot of Emacs customizations and/or
system configuration changes that you didn't describe.

> > Please report any problems with this script to address@hidden .

>
> That's weird because immediately issuing
>
> which automake
>
> says
>
> /usr/bin/automake

Again, for me "which automake" in Eshell says it isn't found, so you
have some non-trivial system configuration there (e.g., your MSYS2 bin
directory is probably on PATH, something that MSYS installation
doesn't do by default, AFAIK). Also, is that 'which' command an MSYS2
program or is it a native Windows program? If the latter, I won't
expect it to use the /usr/bin format, it should show the drive letter.

> I use `bash' as Emacs Shell.

What does that mean, exactly? Which variables did you customize and
how?

> Furthermore, when using `bsdtar' from Emacs Shell, I saw:
>
> bsdtar: Failed to set default locale

That's an unrelated bug in your ported 'bsdtar', I think. The one I
have here works just fine from Eshell command line.

> What could be the problem here?

Not enough info to answer that question. FWIW, once I fixed the
obvious issues with PATH, autogen.sh ran for me without any complaints
inside Eshell.

[Alexander Shukaev]

>
> You mean, Eshell, as in "M-x eshell RET"? Or something else?

​Sorry, I think I was misinterpreted here​. As I said,

I use `bash' as Emacs Shell.

​So it's ​rather "M-x shell".

​​
> An easy way to verify this hunch would be to tell your (Emacs) terminal
> to use "/bin/bash --login" instead of just "/bin/bash" as thedefault shell
> (please substitute /bin/bash by whatever the correct path is in your
> system).
>
> Another thing you could do is to issue in both shells the built-in command
> "set" (which shows you all the environment variable settings) and study
> the differences. Watch out especially for PATH, LANG, LANGUAGE and all the
> other locale stuff, like LC_*
>

Yes, I'm going to try that. One thing I don't like about "--login" though
is that it forces `bash' to `cd' to '~'.

Regards,
Alexander

[Eli Zaretskii]

> Date: Thu, 30 Apr 2015 20:14:08 +0200
> From: Alexander Shukaev <haro...@gmail.com>
> Cc: help-gnu-emacs <help-gn...@gnu.org>

>
> You mean, Eshell, as in "M-x eshell RET"? Or something else?
>
> ​Sorry, I think I was misinterpreted here​. As I said,
>
> I use `bash' as Emacs Shell.
>
> ​So it's ​rather "M-x shell".

That still requires some non-default setup that you didn't describe.

So if you still need help in figuring that out, please provide those
missing details.

[Alexander Shukaev]

>
> That still requires some non-default setup that you didn't describe.
>
> So if you still need help in figuring that out, please provide those
> missing details.
>

​To be honest,​

​I just don't understand what do you mean by non-default setup. I just
start `shell' with 'bash'. My Emacs is already started with MSYS2's
"/usr/bin" and "/mingw64/bin" being available in `PATH'​, so those are also
available inside `shell' (checked). In this case I don't run 'autogen.sh'
directly, I run 'makepkg' and it runs 'autogen.sh' for me. This, however,
should not make any difference because as I said, it works from regular
terminal. I don't get why it would not from Emacs `shell'.

[Eli Zaretskii]

> Date: Thu, 30 Apr 2015 21:44:11 +0200

> From: Alexander Shukaev <haro...@gmail.com>
> Cc: help-gnu-emacs <help-gn...@gnu.org>
>

> That still requires some non-default setup that you didn't describe.
>
> So if you still need help in figuring that out, please provide those
> missing details.
>
> ​To be honest,​
> ​I just don't understand what do you mean by non-default setup. I just start
> `shell' with 'bash'.

How do you do that? In "emacs -Q", "M-x shell" does not invoke Bash,
unless you do something else.

> My Emacs is already started with MSYS2's "/usr/bin" and
> "/mingw64/bin" being available in `PATH'​, so those are also available inside
> `shell' (checked). In this case I don't run 'autogen.sh' directly, I run
> 'makepkg' and it runs 'autogen.sh' for me. This, however, should not make any
> difference because as I said, it works from regular terminal. I don't get why
> it would not from Emacs `shell'.

Neither do I. FWIW, if I do this from MSYS Bash:

./src/emacs -Q
M-x shell RET
bash
./autogen.sh

it runs happily to completion and says I can run './configure'. I
have no idea why it doesn't work for you.

[to...@tuxteam.de]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, Apr 30, 2015 at 08:14:08PM +0200, Alexander Shukaev wrote:
[...]
> > [login shell, check env variables]

> Yes, I'm going to try that. One thing I don't like about "--login" though
> is that it forces `bash' to `cd' to '~'.

Just tried it: bash --login doesn't cd to my $HOME, it just stays "where it is".
Perhaps one of your *profiles (/etc/profile, ~/.profile, ~/.bash_profile or
however those are called in Cygwin-land) is cd-ing to $HOME?

regards

- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlVDRuEACgkQBcgs9XrR2kZB0wCcCBCmzwjRSBZ0vgw+DTDqSk+5
pWEAn1e5EknL1CaN7gymA51pbbFSljy5
=MzOv
-----END PGP SIGNATURE-----

[Alexander Shukaev]

I think I found it in one of "post-install" scripts. Let's see what I can
do about it.

On Fri, May 1, 2015 at 4:33 PM, Alexander Shukaev <haro...@gmail.com>
wrote:

> ​My goodness, this is so weird. I start to hate that, sigh... T_T
>
> So to check that it's indeed "/etc/profile" involved in changing to "~".
> I do the following:
>
> 1. Start 'bash' with "M-x bash RET":
>
> (defun bash
> (&optional buffer)
> (interactive)
> (let ((explicit-shell-file-name "bash")
> (explicit-bash-args '("--noediting" "-i")))
> (setq buffer (my-call shell buffer)))
> (with-current-buffer buffer
> (when (equal (buffer-name) "*shell*")
> (rename-buffer "*bash*" t))))
>
> ​NOTE: `my-call' macro just properly handles whether to
> `call-interactively' or not.​
>
> 2. I have "~/emacs.d/init_bash.sh" with
>
> . /etc/profile
>
> As a result, here is what I see when I start `bash' from some buffer which
> contains file with path let's say "~/dir/file":
>
> 2015.05.01 Friday 16:07:21
> Haroogan@G75VW:~/dir
> $ ]0;~
> Haroogan@G75VW ~
> $
>
> A few points here.
>
> 1. The first prompt is of course mine (coming from ".bashrc").
> 2. Then there goes garbage, it's a separate story by the way. I think I
> discovered a bug: this garbage appears when there is some content
> in "~/emacs.d/init_bash.sh". Does not matter what kind of content that is,
> e.g. it could be even be comments. The main point is that the number of
> those garbage lines that will appear on the start of `bash' is equal to the
> number of lines written in "~/emacs.d/init_bash.sh". Like in this case, we
> just have one line in "~/emacs.d/init_bash.sh", so the garbage appears one
> time. I mean it's like it really types those newlines into shell.
> 3. Finally there is another prompt which is obviously coming from the
> sourced "/etc/profile".
>
> I've scanned "/etc/profile" and other files that it sources. There is no
> evidence of `cd' to "~". I'm attaching it here, can you find anything that
> would change to "~"? Thank you.
>
> ​Regards,
> Alexander​

[to...@tuxteam.de]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, May 01, 2015 at 04:33:04PM +0200, Alexander Shukaev wrote:
> My goodness, this is so weird. I start to hate that, sigh... T_T

I read the rest. Yes, those scripts tend to be somewhat weird.
Your described effects are strange indeed.

But judging by your other post, it seems you are on track: congrats!

regards
- -- t

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlVDlcwACgkQBcgs9XrR2kamGACcDGILPKqdQSq1EwHUHKWbZoix
ZwMAn17+LZjoU8FYTOxcK1wenlIegFRk
=dTW/
-----END PGP SIGNATURE-----

[Alexander Shukaev]

So I found out that the "/etc/post-install/05-home-dir.post" script changes
to `HOME' depending on `CHERE_INVOKING' (sick stuff I must say). Setting

(setenv "CHERE_INVOKING" "1")

solves the problem with unintentional change to `HOME' when "--login" is
present. Now I can finally invoke 'bash' with

(defun bash
(&optional buffer)
(interactive)
(let ((explicit-shell-file-name "bash")

(explicit-bash-args '("--noediting" "-l" "-i")))

(setq buffer (my-call shell buffer)))
(with-current-buffer buffer
(when (equal (buffer-name) "*shell*")
(rename-buffer "*bash*" t))))

So far so good. Now onto the original problem since even with "-l" those
issues with 'bsdtar' and inability to find 'automake' by 'autogen.sh' still
remain. I'm currently looking at the diff between the two `set' commands
from ConEmu and Emacs. There are of course quite some differences, I don't
want to post all of them not to overwhelm you. But here are probably the
most interesting ones:

ConEmu:

LANG=en_US.UTF-8
PKG_CONFIG_PATH=
PROMPT=
SESSIONNAME=
SHELL=/usr/bin/bash

Emacs:

LANG=ENU
PKG_CONFIG_PATH='C:\Tools\x64\MSYS2\mingw64\lib\pkgconfig;'
PROMPT='$P$G'
SESSIONNAME=Console
SHELL=C:/Tools/x64/MSYS2/mingw64/libexec/emacs/25.0.50/x86_64-w64-mingw32/cmdproxy.exe

What do you think guys?

[Alexander Shukaev]

Right, so explicitly doing

(setenv "LANG" "en_US.UTF-8")

solves both problems at once. Interesting... Do you have any comments on
that? Maybe clarifications on why this problem occurs? For instance, why
`LANG' is properly set in ConEmu, while in Emacs it has value `ENU'? Where
does this even come from? Is it OK to do

(setenv "LANG" "en_US.UTF-8")

or there is some other more generic way?

On Fri, May 1, 2015 at 6:08 PM, Alexander Shukaev <haro...@gmail.com>
wrote:

[Eli Zaretskii]

> Date: Fri, 1 May 2015 19:06:09 +0200
> From: Alexander Shukaev <haro...@gmail.com>
> Cc: help-gnu-emacs <help-gn...@gnu.org>, Eli Zaretskii <el...@gnu.org>

>
> Right, so explicitly doing
>
> (setenv "LANG" "en_US.UTF-8")
>
> solves both problems at once. Interesting... Do you have any comments on that?
> Maybe clarifications on why this problem occurs? For instance, why `LANG' is
> properly set in ConEmu, while in Emacs it has value `ENU'?

Because the Windows locale names are different from Posix locale
names.

> Where does this even come from?

>From w32.c:init_environment:

/* Get default locale info and use it for LANG. */
if (GetLocaleInfo (LOCALE_USER_DEFAULT,
LOCALE_SABBREVLANGNAME | LOCALE_USE_CP_ACP,
locale_name, sizeof (locale_name)))
{
for (i = 0; i < N_ENV_VARS; i++)
{
if (strcmp (env_vars[i].name, "LANG") == 0)
{
env_vars[i].def_value = locale_name;
break;

}
}
}

> Is it OK to do
>
> (setenv "LANG" "en_US.UTF-8")

Not if you want the rest of locale-specific code in Emacs itself to
work, no.

Why does a native Windows program such as bsdtar expect a Posix-style
locale spec? It's a bug, IMO. The MS-Windows implementation of
'setlocale' will not work with Posix locale specs.

> or there is some other more generic way?

There's no generic way. Locale names and specifications are not
portable between Posix and non-Posix systems.

[Alexander Shukaev]

I see. Thanks, Eli.

Finally, do you have any comments on the issue
with "~/emacs.d/init_bash.sh" that I've described before. Let me introduce
another test case briefly. Put

# Hello, World!
# Hello, World!
# Hello, World!

into "~/emacs.d/init_bash.sh".

Then start 'bash' from Emacs. Here is what I see:

2015.05.01 Friday 20:30:46
Haroogan@G75VW:~
$
2015.05.01 Friday 20:30:46
Haroogan@G75VW:~
$
2015.05.01 Friday 20:30:46
Haroogan@G75VW:~
$
2015.05.01 Friday 20:30:46
Haroogan@G75VW:~
$

Those 3 newlines were really typed into the shell. How come?

[Eli Zaretskii]

> Date: Fri, 1 May 2015 20:35:11 +0200

> From: Alexander Shukaev <haro...@gmail.com>
> Cc: help-gnu-emacs <help-gn...@gnu.org>
>

> Finally, do you have any comments on the issue with "~/emacs.d/init_bash.sh"
> that I've described before.

Sorry, I don't know anything about init_bash.sh, and until now didn't
even know it existed. Do you know what Emacs feature creates and
maintains it?

[Alexander Shukaev]

>
> ​​
> Sorry, I don't know anything about init_bash.sh, and until now didn't
> even know it existed. Do you know what Emacs feature creates and
> maintains it?
>

​Not sure, but I found this:​


​* Startup Changes in Emacs 22.1

** Init file changes
If the init file ~/.emacs does not exist, Emacs will try
~/.emacs.d/init.el or ~/.emacs.d/init.elc. Likewise, if the shell init file
~/.emacs_SHELL is not found, Emacs will try ~/.emacs.d/init_SHELL.sh.​

​and this:

https://www.gnu.org/software/emacs/manual/html_node/emacs/Interactive-Shell.html#Interactive-Shell
​

[Alexander Shukaev]

Looks like it's here:

(pop-to-buffer buffer)
(unless (comint-check-proc buffer)
(let* ((prog (or explicit-shell-file-name
(getenv "ESHELL") shell-file-name))
(name (file-name-nondirectory prog))
(startfile (concat "~/.emacs_" name))
(xargs-name (intern-soft (concat "explicit-" name "-args"))))
(unless (file-exists-p startfile)
(setq startfile (concat user-emacs-directory "init_" name ".sh")))
(apply 'make-comint-in-buffer "shell" buffer prog
(if (file-exists-p startfile) startfile)
(if (and xargs-name (boundp xargs-name))
(symbol-value xargs-name)
'("-i")))
(shell-mode)))

in the `shell' function.

[Eli Zaretskii]

> Date: Fri, 1 May 2015 21:05:24 +0200

> From: Alexander Shukaev <haro...@gmail.com>
> Cc: help-gnu-emacs <help-gn...@gnu.org>
>

> ​* Startup Changes in Emacs 22.1
>
> ** Init file changes
> If the init file ~/.emacs does not exist, Emacs will try
> ~/.emacs.d/init.el or ~/.emacs.d/init.elc. Likewise, if the shell init file
> ~/.emacs_SHELL is not found, Emacs will try ~/.emacs.d/init_SHELL.sh.​

So you are saying this file should be created and maintained by hand?

If so, what exactly is your problem with it, and why do you need it in
the first place?

[Eli Zaretskii]

> Date: Fri, 1 May 2015 20:35:11 +0200

> From: Alexander Shukaev <haro...@gmail.com>
> Cc: help-gnu-emacs <help-gn...@gnu.org>
>

> Finally, do you have any comments on the issue with "~/emacs.d/init_bash.sh"

> that I've described before. Let me introduce another test case briefly. Put
>
> # Hello, World!
> # Hello, World!
> # Hello, World!
>
> into "~/emacs.d/init_bash.sh".
>
> Then start 'bash' from Emacs. Here is what I see:
>
> 2015.05.01 Friday 20:30:46
> Haroogan@G75VW:~
> $
> 2015.05.01 Friday 20:30:46
> Haroogan@G75VW:~
> $
> 2015.05.01 Friday 20:30:46
> Haroogan@G75VW:~
> $
> 2015.05.01 Friday 20:30:46
> Haroogan@G75VW:~
> $
>
> Those 3 newlines were really typed into the shell. How come?

Now that we know that the contents of this file are sent as an input
string to the shell when it starts, what did you expect? The shell
gets 3 lines, each one of which is a comment, so it does nothing, but
displays the newline. What is surprising here?

[Alexander Shukaev]

The point then is that this file actually does not serve its purpose. It
does not behave as, for example, ".bashrc" does. In other words, one
cannot simply write multiline shell code in there without experiencing this
ugly side effect. I understand the problem here, but why not, for
instance, concatenate lines with " ; "
​​
​in order to make one line out of them​
before sending them to 'bash'?
​

[Eli Zaretskii]

> Date: Fri, 1 May 2015 21:46:53 +0200

> From: Alexander Shukaev <haro...@gmail.com>
> Cc: help-gnu-emacs <help-gn...@gnu.org>
>

> > Those 3 newlines were really typed into the shell. How come?
>
> Now that we know that the contents of this file are sent as an input
> string to the shell when it starts, what did you expect? The shell
> gets 3 lines, each one of which is a comment, so it does nothing, but
> displays the newline. What is surprising here?
>
>
> ​The point then is that this file actually does not serve its purpose. It does
> not behave as, for example, ".bashrc" does. In other words, one cannot simply
> write multiline shell code in there without experiencing this ugly side effect.

I think you can, just with little more creativity (see below), but why
would you want to? You already have .bashrc, and can easily test that
you are running under Emacs, if that's the issue. AFAIU, the
emacs_bash.sh file is supported for some very specialized situations,
and probably for Posix platforms, so I have hard time imagining why it
would be needed on Windows.

> I understand the problem here, but why not, for instance, concatenate lines
> with " ; " ​in order to make one line out of them​ before sending them to 'bash'?

For that, the code needs to be much smarter and understand the
semantics of the string it sends. E.g., what if a line ends with a
backslash, or some other special character?

But you can do that yourself -- make only one line out of several.