Go app using sshclient to mount directories using sshfs - mountpoints fail to persist

[Piotr Chudzik]

Hello there!

I am trying to create a go application that would run a docker container and mount a host directory inside a container using sshfs( rejected using volumes because they mess up directory ownership, btw if anyone is using docker and might know this:can you create a file inside a mounted directory using volumes which ownership can be somehow specified? So it does not have to be "root created" but specific suer created? sshfs has "idmap=user" flag which is great, is there something analogical in docker?)

So I create a daemonized container:
sudo docker run -i -t -privileged -dns=[172.25.0.10] -p 22 -d orobix/sshfs_startup_key2 /bin/bash -c "/usr/sbin/sshd -D"

And then I can ssh into it and mount directories using sshfs(which is preinstalled in the image):

sshfs pi...@172.17.42.1:/home/piotr/helloworld/ /mnt -o idmap=user

And when I ssh out of it (type exit) and ssh back into it, the mountpoint is still there which is great and desired behaviour.

However when I execute the same steps using Go and crypto/ssh client it will create create mountpoints using sshfs, but after the Go program terminates the mountpoint dissapears and I do not know why...

From my perspective I am recreating the same steps in go code as i would do it manually but for some reason the mountpoint disappears..
I tried using sshfs "-o reconnect" option inside the go code which seems to keep the mountpoints inside the container(after typing "mount" command inside the container I can see that the mountpoint is there:

root@bd901e2:~# mount
none on / type aufs (rw,relatime,si=d49df04d58206c91)
/dev/disk/by-uuid/7e1d6bab-b3f2-4ac3-8bff-0779f5bf40f2 on /etc/hostname type ext4 (ro,relatime,errors=remount-ro,data=ordered)
/dev/disk/by-uuid/7e1d6bab-b3f2-4ac3-8bff-0779f5bf40f2 on /etc/hosts type ext4 (ro,relatime,errors=remount-ro,data=ordered)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
/dev/disk/by-uuid/7e1d6bab-b3f2-4ac3-8bff-0779f5bf40f2 on /.dockerinit type ext4 (ro,relatime,errors=remount-ro,data=ordered)
/dev/disk/by-uuid/7e1d6bab-b3f2-4ac3-8bff-0779f5bf40f2 on /etc/resolv.conf type ext4 (ro,relatime,errors=remount-ro,data=ordered)
devpts on /dev/tty1 type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=666)
devpts on /dev/ptmx type devpts (rw,relatime,mode=600,ptmxmode=666)
pi...@172.17.42.1:/home/piotr/helloworld/ on /mnt type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,max_read=65536)

However when I try to access it in any way I am getting errors:

root@bd901e2:~# cd /mnt/
root@bd901e2:/mnt# ls
ls: reading directory .: Input/output error

This is my sshclient code:

package main

import (
//"bytes"
"code.google.com/p/go.crypto/ssh"
//"fmt"
"io"
"log"
"os"
)

var (
server = "172.17.42.1:49155"
username = "root"
password = clientPassword("orobix2013")
)

type clientPassword string

func (p clientPassword) Password(user string) (string, error) {
return string(p), nil
}

type TerminalModes map[uint8]uint32

const (
VINTR = 1
VQUIT = 2
VERASE = 3
VKILL = 4
VEOF = 5
VEOL = 6
VEOL2 = 7
VSTART = 8
VSTOP = 9
VSUSP = 10
VDSUSP = 11
VREPRINT = 12
VWERASE = 13
VLNEXT = 14
VFLUSH = 15
VSWTCH = 16
VSTATUS = 17
VDISCARD = 18
IGNPAR = 30
PARMRK = 31
INPCK = 32
ISTRIP = 33
INLCR = 34
IGNCR = 35
ICRNL = 36
IUCLC = 37
IXON = 38
IXANY = 39
IXOFF = 40
IMAXBEL = 41
ISIG = 50
ICANON = 51
XCASE = 52
ECHO = 53
ECHOE = 54
ECHOK = 55
ECHONL = 56
NOFLSH = 57
TOSTOP = 58
IEXTEN = 59
ECHOCTL = 60
ECHOKE = 61
PENDIN = 62
OPOST = 70
OLCUC = 71
ONLCR = 72
OCRNL = 73
ONOCR = 74
ONLRET = 75
CS7 = 90
CS8 = 91
PARENB = 92
PARODD = 93
TTY_OP_ISPEED = 128
TTY_OP_OSPEED = 129
)

func main() {
// An SSH client is represented with a slete). Currently only
// the "password" authentication method is supported.
//
// To authenticate with the remote server you must pass at least one
// implementation of ClientAuth via the Auth field in ClientConfig.

config := &ssh.ClientConfig{
User: username,
Auth: []ssh.ClientAuth{
// ClientAuthPassword wraps a ClientPassword implementation
// in a type that implements ClientAuth.
ssh.ClientAuthPassword(password),
},
}
client, err := ssh.Dial("tcp", "172.17.42.1:49155", config)
if err != nil {
panic("Failed to dial: " + err.Error())
}

// Each ClientConn can support multiple interactive sessions,
// represented by a Session.
defer client.Close()
// Create a session
session, err := client.NewSession()
if err != nil {
log.Fatalf("unable to create session: %s", err)
}
defer session.Close()
// Set up terminal modes
modes := ssh.TerminalModes{
ECHO: 0, // disable echoing
TTY_OP_ISPEED: 14400, // input speed = 14.4kbaud
TTY_OP_OSPEED: 14400, // output speed = 14.4kbaud
}
// Request pseudo terminal
if err := session.RequestPty("xterm", 80, 40, modes); err != nil {
log.Fatalf("request for pseudo terminal failed: %s", err)
}
//var b bytes.Buffer
//session.Stdout = &bi

stdin, _ := session.StdinPipe()

stdout, _ := session.StdoutPipe()

go io.Copy(os.Stdout, stdout)
go io.Copy(stdin, os.Stdin)
//go io.Copy(os.Stderr, stderr)
if err := session.Run("/bin/bash -c \"sshfs pi...@172.17.42.1:/home/piotr/helloworld/ /mnt -o idmap=user;touch /mnt/ofoo\""); err != nil {
panic("Failed to run: " + err.Error())
}

After I run this code the "ofoo" file will be created on host which is a proof that sshfs worked correctly. However when I ssh into the container, the mountpoints is gone and I would like it to stay there until the container is killed/stoped. I do not have this issue when doing everything manually. So how I can make the mountpoint persistent?

[Piotr Chudzik]

Bump.  Any suggestions/wild thoughts why golang app reproducing manual steps(which work) fails to work with sshfs are welcome!

[Dave Cheney]

try adding -x -e to your bash invocation.

On Thu, Oct 17, 2013 at 1:05 AM, Piotr Chudzik <piotrch...@gmail.com> wrote:
> Bump. Any suggestions/wild thoughts why golang app reproducing manual
> steps(which work) fails to work with sshfs are welcome!
>

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Piotr Chudzik]

Thanks Dave for your reply!

Unfortunately adding  -x -e to  the bash invocation does not help, the mountpoints are still disappearing after the go program finishes:

if err := session.Run("/bin/bash -x -e -c \"sshfs pi...@172.17.42.1:/home/piotr/helloworld/ /mnt/ -o idmap=user;touch /mnt/aaa;/usr/sbin/sshd\""); err != nil {

    panic("Failed to run: " + err.Error())

  }

The difference now is that it shows what commands it runs:

 piotr@saphira:~/go/src/github.com/specialp89/crane$ go run sshClient.go

+ sshfs pi...@172.17.42.1:/home/piotr/helloworld/ /mnt/ -o idmap=user

pi...@172.17.42.1's password: 

+ touch /mnt/aaa

+ /usr/sbin/sshd

So the commands are executed but why the mountpoints disappear after go program finishes? Is it something to do with the fact that go program does:

 session.Close()

?

Any input is very welcome, I spent almost two weeks now on this issue without progress so EVERY idea is very much welcome:)

[Jonathan Pittman]

I would imagine that something is getting killed when the session ends.

Do you have to do /bin/bash -x -e -c "<command snippet>" ?

What happens if you do this?

session.Run("sshfs pi...@172.17.42.1:/home/piotr/helloworld/ /mnt/ -o idmap=user && touch /mnt/aaa && /usr/sbin/sshd &")

[Piotr Chudzik]

W dniu środa, 23 października 2013 21:24:26 UTC+2 użytkownik Jonathan Pittman napisał:

I would imagine that something is getting killed when the session ends.

Do you have to do /bin/bash -x -e -c "<command snippet>" ?

I don't have to do this (there is no particular reason) but this is a nice way (-c) to pass multiple commands so I used it. 

What happens if you do this?

session.Run("sshfs pi...@172.17.42.1:/home/piotr/helloworld/ /mnt/ -o idmap=user && touch /mnt/aaa && /usr/sbin/sshd &")

It doesn't throw an error but it doesn't ask me for the host password to access the "helloworld" directory so I guess that it fails quietly somewhere along the lines (no error message unfortunately). Interestingly enough dropping last "&" makes it work as my initial code:

session.Run("sshfs pi...@172.17.42.1:/home/piotr/helloworld/ /mnt/ -o idmap=user && touch /mnt/aaa && /usr/sbin/sshd ")

It asks for host password, creates the "aaa" file but once the golang app finishes the mountpoint inside the container is gone.

So using "-c" and && seems to produce the same results here.

Any idea what might be getting killed or how I can track it down?

[Kevin Gillette]

It may be that sshfs is expecting to be directly attached to a tty; you can simulate a tty without actually being one, but it takes a bit of work.

[Piotr Chudzik]

I thought that the pseudo terminal the ssh client is using is enough. After all it successfully is setting up the ssh connection and the commands are executed, is there a big difference between using pty and tty?

[Jonathan Pittman]

session.Run does not allocate a tty or pty as far as I know.  session.Run is just like (or should be like) doing this...

ssh user@host "sshfs pi...@172.17.42.1:/home/piotr/helloworld/ /mnt/ -o idmap=user; touch /mnt/aaa; /usr/sbin/sshd"

I looked back at your code (it helps to use the go playground and the share feature instead of pasting it all in email).  You do not need to request the pty if you are going use Run.  If you were planning to use Shell, then requesting a pty makes sense.

The use of "&&" vs "-c" does not carry a useful distinction.  The "-c" is a flag for bash to "run this command."  The "&&" is more like ";" in that it is used to separate commands.  Specifically, the use of ";" between commands means "do this and then do this" regardless of the success or failure of the previous command.  The use of "&&" means "do this, if that was successful then do this."  Similarly "||" would be "do this, if that fails do this."

I suggested the use of "&&" to see if any of the first part of the command sequence might be failing.  The single "&" was simply to background the process on the host to see if it would keep running after the session closed (in case that was related to the problem).

Try this code.  http://play.golang.org/p/pGfBNINnwn

This may be no better since it is similar to what you are doing now.  As Kevin indicated, "It may be that sshfs is expecting to be directly attached to a tty."

Do you need/want this to work with session.Run or can you request a pty and then use session.Shell and type out the commands the way you indicated with regular ssh?

[Piotr Chudzik]

Thanks for the code example @Jonathan!

I tried  using your code snippet but now it did not even try to ask me for the host password so I assume that the first sshfs command wasn't even executed, I think that the reason it didn't work is because the pseudo tty code was removed. Once I add it back it executes all commands without a problem. Adding pty code makes it pretty much the same as my initial code (apart not using -c option) so no surprise here.

Based on this I think that allocating pty even when using session.Run is necessary here. I tried to use session.Using either Run or Shell is fine with me as longest I can automate the sshfs mounting process as much as possible. I tried using Shell and pty but nothing happens when I execute this code:

http://play.golang.org/p/eDW1sVoafQ

Most likely I don't use session.Shell in the correct way, what do you think?

I thought that using pty removes the need of using terminal for sshfs but if that is the problem (I genuinely running out of ideas here) I am fine with using terminal, however how I can go about it?

I uploaded my testing docker container to the public repository so you can run and test it now yourself:

First you need to start the container in the background with sshd running:

sudo docker run -i -t -privileged -dns=172.25.0.10 -p 22 -d orobix/sshfs_startup_key2 /bin/bash -c "/usr/sbin/sshd -D"

privileged is needed for the sshfs fuse system to work, I use dns option to specify my dns server because I am inside a local network(you might not need to)

The image(orobix/ssfs_startuo_key2) should be pulled automatically from the public repository.

Once the container is running you should be able to run the go code(of course ip addresses must be changed).

You can ssh into container manually with e.g.:

ssh ro...@172.17.42.1 -p 49153

Again, ip and ports will be different.

[Jonathan Pittman]

Here is an example using RequestPty and Shell.

http://godoc.org/code.google.com/p/go.crypto/ssh#example-Session-RequestPty

I would actually really like to test this out myself to dig into the issue, but I am unable to look at this in detail for the next few weeks.  Maybe someone else on this list will have more time sooner.

[Piotr Chudzik]

Here is an example using RequestPty and Shell.

http://godoc.org/code.google.com/p/go.crypto/ssh#example-Session-RequestPty

That's exactly the same example I used which does not work.... 

I would actually really like to test this out myself to dig into the issue, but I am unable to look at this in detail for the next few weeks.  Maybe someone else on this list will have more time sooner.

Could you at least point me in the right direction please?