Shell commands in Julia 0.4.2 in Windows 10

[Jason Bates]

Hi All,

I am having issues executing shell commands from within Julia (either in the REPL or via the iJulia notebook) on Windows 10.  I can run actual programs from within Julia, but all attempts at executing shell commands produce an error.

For instance, the example from the documentation:

julia> run(`echo hello`)

produces

julia> run(`echo hello`)
ERROR: could not spawn `echo hello`: no such file or directory (ENOENT)
 in _jl_spawn at process.jl:262

I get the same result using the semi-colon shortcut, both in the REPL and within iJulia.

However, I am able to call windows programs from within Julia successfully.  For instance, 

julia> run(`notepad`)

successfully spawns a notepad window.

Has anyone else run into this or similar issues on Win10?  Google searches have turned up empty.  Further, the issue appears to be exclusive to Julia; shell magic commands within iPython still execute correctly.

Thanks,

Jason Bates

[ele...@gmail.com]

IIUC Julia does not use a shell, so it cannot run any command that is built-in to a shell.  On Linux "echo" is a program, not a built-in but maybe its different on windows.

[Tony Kelman]

You can manually go through a shell via `cmd /C echo hello`, or better (slower to start but much more sane and useful) `powershell -Command "echo hello"`

[hustf]

I for one really miss the shell commands, which disappeared with 0.4.2. 

Fortunately, cd still works, but that's about it. pwd() inside of Julia is not so bad, but you soon tire of typing readdir()  instead of just ;ls. If there are more than a few files in the directory, the output is truncated, too. For new users testing Julia from the documentation this gives a pretty bad impression.

Fortunately, this seems to be a temporary state of affairs, and with the new git library integrated with Julia, the package system will be a lot faster, too. There is an explanation in the thread below. Just don't follow the (my) advice and cut down your PATH. If you meet problems with it, try rearranging the PATH variable or put the julia and atom paths in the "system" and not "user" part of the definition. You need to be logged in to an administrator account to change the system PATH definitions, I believe.

You will likely see less issues than I did with the PATH since Juno/ Atom/ Lighttable (or whatever it's really called) is now updated and gives more flexibility.

https://groups.google.com/forum/#!searchin/julia-users/powershell/julia-users/pHonnw8X3iw/Vcbuj5amAwAJ 

[Tony Kelman]

Unix shell commands that we happened to be getting from git are no longer on Julia's path. This is not temporary. Windows is not unix. If you want to use unix shell commands on windows, either download and add your own windows posix layer to your path (temporarily in juliarc, not permanently where it will break other software), or call powershell which has many aliases spelled the same as unix shell commands.

[Páll Haraldsson]

On Wednesday, January 13, 2016 at 3:36:08 AM UTC, Jason Bates wrote: 

Further, the issue appears to be exclusive to Julia; shell magic commands within iPython still execute correctly.

Just to be sure, do you mean iPython with Python? Or Jupyther a fork of it, that works with Julia (and python)?

In case you are saying this or something, works better in [i]Python, then I would like feature parity, BUT:

I support not using a shell, e.g. bash, as that would add to the footprint of Julia.. (and bash is getting old/outdated and see recent Shellshock security issue..).

This advise may not be helpful..:

>You can manually go through a shell via `cmd /C echo hello`, or better (slower to start but much more sane and useful) `powershell -Command "echo hello"

As I REALLY like how Julia is cross-platform, and I hope people will not start hardcoding such in their code.. In case Windows variants, allow the Unix "shell magic commands" as aliases, it's regrettable that they can't be used, without adding platform-specific code.. Would clearly specified NON-CROSS-PLATFORM, shell() in addition to run() be helpful (should it run cmd or powershell on Windows?) as in some cases it's actually more portable..? Maybe in a package.. 

 

Thanks,

Jason Bates

[Tony Kelman]

Another useful option is called busybox-w32. It's a small ~400kb exe that works like busybox on linux, where you can call it as many different coreutils functions. The shell it implements is ash rather than full featured bash, but posix shell scripts that don't use bashisms should run fine.

We used to include a copy of busybox-w32 with julia for the sake of passing some of the spawn tests, but took it out as it's not necessary with git 2.x. Busybox is GPL licensed so it's probably better to put in a package.

[Jason Bates]

Unix shell commands that we happened to be getting from git are no longer on Julia's path. This is not temporary. Windows is not unix. If you want to use unix shell commands on windows, either download and add your own windows posix layer to your path (temporarily in juliarc, not permanently where it will break other software), or call powershell which has many aliases spelled the same as unix shell commands.

This is disappointing.  I understand the compatibility issues raised above, but losing this feature on windows is not a change for the better.  I'm not asking windows to be unix, but access to shell commands of some sort would be nice.  There's very little difference between ;ls and ;dir, but there is a significant different between ;dir and run(`cmd dir`).

Just to be sure, do you mean iPython with Python? Or Jupyther a fork of it, that works with Julia (and python)?

Specifically, I am referring to a Jupyter notebook running an Anaconda Python 3.5 kernel, versus that same notebook running a Julia 0.4.2 kernel on Windows 10.  With the Python kernel active, the shell magic command %ls works.  With the Julia kernel running, ;ls fails (as does ;dir, or any other call to a shell command).

Jason

[Tony Kelman]

If you can suggest a non-GPL-licensed method of obtaining coreutils, I'd be open to incorporating it somehow. I'm planning on putting together a few packages (where GPL isn't quite as much of an issue) that provide convenience wrappers around busybox-w32 or powershell, but haven't devoted much time to it yet.

A temporary measure you can do is add ENV["PATH"] *= ";" * joinpath(JULIA_HOME,"..","Git","usr","bin")

in your juliarc. This will prevent Julia from starting inside Cygwin or MSYS2, which is why I'm reluctant to make this default behavior.

(was missing the semicolon in the suggested code snippet first time around, sorry)

[Andreas Lobinger]

Hello colleague,

On Wednesday, January 13, 2016 at 11:28:39 PM UTC+1, Tony Kelman wrote:

Unix shell commands that we happened to be getting from git are no longer on Julia's path. This is not temporary. Windows is not unix. If you want to use unix shell commands on windows, either download and add your own windows posix layer to your path (temporarily in juliarc, not permanently where it will break other software), or call powershell which has many aliases spelled the same as unix shell commands.

i agree, windows is not unix (it never was) and the following might not be helpful, but
Matlab provides (on any OS) a very small set of shell-like commands (cd, dir, ls,mkdir, rmdir,  delete, movefile, copyfile).

And providing something like this at a julia REPL should be considered user friendly.

Wishing a happy day,
       Andreas
 

[Páll Haraldsson]

On mið 20.jan 2016 12:56, Andreas Lobinger wrote:
> Hello colleague,
>
> On Wednesday, January 13, 2016 at 11:28:39 PM UTC+1, Tony Kelman wrote:
>
> Unix shell commands that we happened to be getting from git are no
> longer on Julia's path. This is not temporary. Windows is not unix.
> If you want to use unix shell commands on windows, either download
> and add your own windows posix layer to your path (temporarily in
> juliarc, not permanently where it will break other software), or
> call powershell which has many aliases spelled the same as unix
> shell commands.
>
>
> i agree, windows is not unix (it never was) and the following might not
> be helpful, but
> Matlab provides (on any OS) a very small set of shell-like commands (cd,
> dir, ls,mkdir, rmdir, delete, movefile, copyfile).

Just to be clear, is this issue about shell access in the REPL, or for
run() ?

If only in the REPL, as for convenience, for interactive use, I see not
problem with accessing the cmd or powershell (latter better? does it or
former have a good subset of compatibility, e.g. with the subset above?)
on Windows and bash in Linux/Unix for e.g.:

for i in $( ls ); do
echo item: $i
done

that didn't work, and I wouldn't expect to work in Windows (but I would
in Unix).

> And providing something like this at a julia REPL should be considered
> user friendly.

Exactly, but only in the REPL. If run() would allow access to a shell,
then incompatibility comes in, in full force or we would promise to
provide one (e.g. bash) for Unix compatibility - and it would be a bad idea.

--
Palli.

[Tony Kelman]

Julia has portable functions for all of these, you don't need to use shell mode for cd, dir, readdir, mkdir, rm, mv, or cp.

[Yichao Yu]

On Wed, Jan 20, 2016 at 10:39 AM, Tony Kelman <to...@kelman.net> wrote:
> Julia has portable functions for all of these, you don't need to use shell
> mode for cd, dir, readdir, mkdir, rm, mv, or cp.

FWIW, you can't use shell for `cd` on any platforms supported by julia.

[J Luis]

On Windows one can

julia> pwd()
"C:\\programs\\Julia-0.4"

shell> cd
C:\j

julia> pwd()
"C:\\j"

[Tom Short]

I use `cd` under shell on Linux, too. It's a nice shortcut.