AUTOMATE SFTP USING PEXPECT IN YAML SCRIPT NOT WORKING...

[Zimidar Boy]

Hi guys,

              I am working on project to in which I am trying to backup some files from one server and  than transfer that files to NAS terastation. So from my ansible box i do ssh into node. Then i make backup of some files at same node and create sftp session to transfer files to NAS and i am putting current day date and host name as name of files. The problem is that this  (sftp <ip>)  command prompts me to enter password and to enter password i am trying to build a script which automatically enters password and continue the process. Even i have no error when i ran my playbook but i don't get my files at tera station. But when i ran my playbook with level 1 verbose like -v it shows something like this but in yellow colour so not like error:

TASK [Creates Sftp session to send files to terastation.] ****************************************************************************************************************************************************************************************************************************

changed: [localhost] => {"changed": true, "cmd": "sftp user@hostname", "delta": "0:00:01.136744", "end": "2017-10-11 05:14:25.224441", "failed": false, "rc": 0, "start": "2017-10-11 05:14:24.087697", "stdout": "Connecting to terastation...\r\nPassword: \r\nsftp> exit", "stdout_lines": ["Connecting to terastation...", "Password: ", "sftp> exit"]}

Here is sample of my script:

- hosts: localhost

  sudo: true

  tasks:

   - name: Log  into node

     shell: ssh user@hostname

   - name: make backup of files

     shell: tar cvf "$HOSTNAME"_`date +"%Y-%m-%d"`.tar /etc/ansible/playbooks

   - name: Creates Sftp session to send files to terastation.

     expect:

       command: sftp user@hostname

       responses:

           (?i)Password: "1rfr2345"

           (?i)sftp>: "cd abc/def/"

           (?i)sftp>: "put "$HOSTNAME"_`date +"%Y-%m-%d"`.tar  "

           (?i)sftp>: "exit"

           

If more info needed pls let me know. Thanks in advance.

[Kai Stian Olstad]

On 12.10.2017 15:29, Zimidar Boy wrote:
> TASK [Creates Sftp session to send files to terastation.]
> ****************************************************************************************************************************************************************************************************************************
> changed: [localhost] => {"changed": true, "cmd": "sftp user@hostname",
> "delta": "0:00:01.136744", "end": "2017-10-11 05:14:25.224441",
> "failed":
> false, "rc": 0, "start": "2017-10-11 05:14:24.087697", "stdout":
> "Connecting to terastation...\r\nPassword: \r\nsftp> exit",
> "stdout_lines":
> ["Connecting to terastation...", "Password: ", "sftp> exit"]}

It do login but just run and exit afterward, and exit is the last of you
responses.

> Here is sample of my script:
>
> - hosts: localhost
> sudo: true
> tasks:
>
> - name: Log into node
> shell: ssh user@hostname
>
> - name: make backup of files
> shell: tar cvf "$HOSTNAME"_`date +"%Y-%m-%d"`.tar
> /etc/ansible/playbooks
>
>
> - name: Creates Sftp session to send files to terastation.
> expect:
> command: sftp user@hostname
> responses:
> (?i)Password: "1rfr2345"
> (?i)sftp>: "cd abc/def/"
> (?i)sftp>: "put "$HOSTNAME"_`date +"%Y-%m-%d"`.tar "
> (?i)sftp>: "exit"
>
>
> If more info needed pls let me know. Thanks in advance.

I think you only can have one unique question with the responses in a
list like this.

- name: Creates Sftp session to send files to terastation.
expect:
command: sftp user@hostname
responses:
(?i)Password: "1rfr2345"
(?i)sftp>:

- "cd abc/def/"
- "put "$HOSTNAME"_`date +"%Y-%m-%d"`.tar "
- "exit"

--
Kai Stian Olstad

[Zimidar Boy]

Thanks for help i have another question so as i am running script as localhost it is making backup on local host even though in ssh into another node i tried by combining backup and ssh process together but it didn't worked error says no such file directory.

  - name: make backup

     shell: |

         ssh user@hostname

         tar cvf "$HOSTNAME"_`date +"%Y-%m-%d"`.tar /opt/backup_test

error:

 TASK [make backup] **************************************************************************************************************************************************************************************************************************

fatal: [localhost]: FAILED! => {"changed": true, "cmd": "ssh user@hostname \n tar cvf \"$HOSTNAME\"_`date +\"%Y-%m-%d\"`.tar /opt/backup_test", "delta": "0:00:00.077642", "end": "2017-10-12 02:07:50.314514", "failed": true, "msg": "non-zero return code", "rc": 2, "start": "2017-10-12 02:07:50.236872", "stderr": "Pseudo-terminal will not be allocated because stdin is not a terminal.\r\ntar: Removing leading `/' from member names\ntar: /opt/backup_test: Cannot stat: No such file or directory\ntar: Exiting with failure status due to previous errors", "stderr_lines": ["Pseudo-terminal will not be allocated because stdin is not a terminal.", "tar: Removing leading `/' from member names", "tar: /opt/backup_test: Cannot stat: No such file or directory", "tar: Exiting with failure status due to previous errors"], "stdout": "", "stdout_lines": []}

PLAY RECAP **********************************************************************************************************************************************************************************************************************************

localhost                  : ok=1    changed=0    unreachable=0    failed=1

[Dan Linder]

The error message really looks like you don't have "/opt/backup_test" on your local system or possibly not writable by the account Ansible is using.

If you have Ansible 2.3 and you're just building a tar file, why not use the "archive" module?

    http://docs.ansible.com/ansible/latest/archive_module.html

 - name: make backup using archive module
   archive:
     dest:"{{ansible_hostname}}_{{ansible_facts.date}}.tar.gz"
     path: /opt/backup_test
     format: gz

DanL

[Zimidar Boy]

Thanks buddy it helped a lot. Appreciated. 

[Zimidar Boy]

Hello Kai,

                     As soon as i used your solution now i am getting this error:

  "msg": "No remaining responses for '(?i)sftp>', output was ': \r\nsftp>'"

On Thursday, October 12, 2017 at 9:42:06 AM UTC-4, Kai Stian Olstad wrote:

On 12.10.2017 15:29, Zimidar Boy wrote:
> TASK [Creates Sftp session to send files to terastation.]
> ****************************************************************************************************************************************************************************************************************************
> changed: [localhost] => {"changed": true, "cmd": "sftp user@hostname",
> "delta": "0:00:01.136744", "end": "2017-10-11 05:14:25.224441",
> "failed":
> false, "rc": 0, "start": "2017-10-11 05:14:24.087697", "stdout":
> "Connecting to terastation...\r\nPassword: \r\nsftp> exit",
> "stdout_lines":
> ["Connecting to terastation...", "Password: ", "sftp> exit"]}

It do login but just run and exit afterward, and exit is the last of you
responses.

> my code

   - name: Creates Sftp session to send files to terastation.
     expect:

       command: ssh user@hostname
       responses:

             (?i)root@": "sftp user@hostname"

[Kai Stian Olstad]

On 19. okt. 2017 17:02, Zimidar Boy wrote:
>
> Hello Kai,
> As soon as i used your solution now i am getting this
> error:
>
> "msg": "No remaining responses for '(?i)sftp>', output was ': \r\nsftp>'"
>

Without the full verbose output it's difficult to say.

But some systems print out the prompt after exit and before the command terminates.
This will trigger this message in Ansible.
You could try adding a extra exit, if that doesn't help you can set ignore_errors: true or write a expect script and run that instead.


--
Kai Stian Olstad

[Zimidar Boy]

Hello Kai,

                   Solved the issue but i have to add 5 exit i don't know why. Do you think it has to do something with number of commands as i am running 5 commands. new code:

- name: crates sftp session to terastation

    expect:

       command: ssh user@host

       responses:

           (?i)root@: ['sftp user@host', 'mkdir /tmp/backups', 'mv lst*tar /tmp/backups', 'cd /tmp/backups', 'rm -rf $(ls -1t /tmp/backups | tail -n +3)', 'exit', 'exit', 'exit', 'exit',' exit']

           (?i)Password: 213gfdshaa

           (?i)sftp>: ['cd a/o/h', 'put lst*tar', 'exit']

[Zimidar Boy]

On Thursday, October 19, 2017 at 2:39:52 PM UTC-4, Zimidar Boy wrote:

Hello Kai,

                   Solved the issue but i have to add 5 exit i don't know why. Do you think it has to do something with number of commands as i am running 5 commands for 1st question.

[Kai Stian Olstad]

On 19. okt. 2017 20:39, Zimidar Boy wrote:
>
> On Thursday, October 19, 2017 at 12:44:16 PM UTC-4, Kai Stian Olstad wrote:
>>
>> On 19. okt. 2017 17:02, Zimidar Boy wrote:
>>>
>>> Hello Kai,
>>> As soon as i used your solution now i am getting
>> this
>>> error:
>>>
>>> "msg": "No remaining responses for '(?i)sftp>', output was ':
>> \r\nsftp>'"
>>>

> Solved the issue but i have to add 5 exit i don't know
> why. Do you think it has to do something with number of commands as i am
> running 5 commands. new code:
>

I you previous message it says

"No remaining responses for '(?i)sftp>'"

So adding exit on the "root@" prompt shouldn't have any effect.

> - name: crates sftp session to terastation
> expect:
> command: ssh user@host
>
>
> responses:
> (?i)root@: ['sftp user@host', 'mkdir /tmp/backups', 'mv lst*tar
> /tmp/backups', 'cd /tmp/backups', 'rm -rf $(ls -1t /tmp/backups | tail -n
> +3)', 'exit', 'exit', 'exit', 'exit',' exit']
>
> (?i)Password: 213gfdshaa
> (?i)sftp>: ['cd a/o/h', 'put lst*tar', 'exit']
>

To understand what is happening you would need to provide a verbose output by running but

ANSIBLE_STDOUT_CALLBACK=debug ansible-playbook -vvv your-playbook.yml



--
Kai Stian Olstad

[Zimidar Boy]

Hey Kai,

               First of all thanks for help. here is output:

changed: [localhost] => (item=hurry-up-1) => {

    "changed": true,

    "cmd": "ssh \"hurry-up-1\"",

    "delta": "0:00:00.382495",

    "end": "2017-10-19 15:18:03.301177",

    "failed": false,

    "invocation": {

        "module_args": {

            "chdir": null,

            "command": "ssh \"hurry-up-1\"",

            "creates": null,

            "echo": false,

            "removes": null,

            "responses": {

                "(?i)root": "tar cvf \"$HOSTNAME\"_`date +\"%Y-%m-%d_%H.%M.%S\"`.tar  test\nexit\n"

            },

            "timeout": 30

        }

    },

    "item": "hurry-up-1",

    "rc": 0,

    "start": "2017-10-19 15:18:02.918682"

}

STDOUT:

Last login: Thu Oct 19 14:33:19 2017 from localhost

hurry-up-1 ~]# tar cvf "$HOSTNAME"_`date +"%Y-%m-%d_%H.%M.%S"`.tar

test/

[root@hurry-up-1 ~]# exit

logout

Connection to hurry-up-1 closed.

TASK [setup-sftp-session : crates sftp session to terastation] ******************************************************************************************************************************************************************************

task path: /etc/ansible/roles/setup-sftp-session/tasks/setup-sftp.yml:13

Using module file /usr/lib/python2.6/site-packages/ansible-2.4.1.0-py2.6.egg/ansible/modules/commands/expect.py

<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root

<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'

<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1508440683.35-276654372992574 `" && echo ansible-tmp-1508440683.35-276654372992574="` echo /root/.ansible/tmp/ansible-tmp-1508440683.35-276654372992574 `" ) && sleep 0'

<127.0.0.1> PUT /tmp/tmpTC63A9 TO /root/.ansible/tmp/ansible-tmp-1508440683.35-276654372992574/expect.py

<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1508440683.35-276654372992574/ /root/.ansible/tmp/ansible-tmp-1508440683.35-276654372992574/expect.py && sleep 0'

<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python2.6 /root/.ansible/tmp/ansible-tmp-1508440683.35-276654372992574/expect.py; rm -rf "/root/.ansible/tmp/ansible-tmp-1508440683.35-276654372992574/" > /dev/null 2>&1 && sleep 0'

changed: [localhost] => {

    "changed": true,

    "cmd": "ssh 192.168.3.40",

    "delta": "0:00:01.737366",

    "end": "2017-10-19 15:18:05.204508",

    "failed": false,

    "invocation": {

        "module_args": {

            "chdir": null,

            "command": "ssh 192.168.3.40",

            "creates": null,

            "echo": false,

            "removes": null,

            "responses": {

                "(?i)Password": "jghsjd674783",

                "(?i)root@": [

                    "sftp test...@192.168.3.43",

                    "mkdir /tmp/backups",

                    "mv ltt-new*tar /tmp/backups",

                    "cd /tmp/backups",

                    "rm -rf $(ls -1t /tmp/backups | tail -n +3)",

                    "exit",

                    "exit",

                    "exit",

                    "exit",

                    " exit"

                ],

                "(?i)sftp>": [

                    "cd a/b/c",

                    "put ltt-new*tar",

                    "exit"

                ]

            },

            "timeout": 30

        }

    },

    "rc": 0,

    "start": "2017-10-19 15:18:03.467142"

}

STDOUT:

Last login: Thu Oct 19 15:18:02 2017 from localhost

[root@hurry-up-1 ~]# sftp test...@192.168.3.43

Connecting to 192.168.3.43...

mkdir /tmp/backups

Password:

sftp> cd a/b/c

sftp> put ltt-new*tar

Uploading hurry-up-1_2017-10-19_15.18.03.tar to /mnt/a/b/c/hurry-up-1_2017-10-19_15.18.03.tar

hurry-up-1_2017-10-19_15.18.03.ta 100%   10KB  10.0KB/s   00:00

sftp> exit

[root@hurry-up-1 ~]# mv ltt-new*tar /tmp/backups

[root@hurry-up-1 ~]# cd /tmp/backups

)root@hurry-up-1 backups]# rm -rf $(ls -1t /tmp/backups | tail -n +3

[root@hurry-up-1 backups]# exit

logout

Connection to 192.168.3.40 closed.

[Kai Stian Olstad]

On 19.10.2017 21:28, Zimidar Boy wrote:
> "command": "ssh \"hurry-up-1\"",
> "creates": null,
> "echo": false,
> "removes": null,
> "responses": {
> "(?i)root": "tar cvf \"$HOSTNAME\"_`date
> +\"%Y-%m-%d_%H.%M.%S\"`.tar test\nexit\n"
> },
> "timeout": 30
> }
> },
> "item": "hurry-up-1",
> "rc": 0,
> "start": "2017-10-19 15:18:02.918682"
> }
>
> STDOUT:
>
> Last login: Thu Oct 19 14:33:19 2017 from localhost
> hurry-up-1 ~]# tar cvf "$HOSTNAME"_`date +"%Y-%m-%d_%H.%M.%S"`.tar
> test/
> [root@hurry-up-1 ~]# exit
> logout
> Connection to hurry-up-1 closed.

Just a tip, in this task you don't need to use the expect module.
If the host hurry-up-1 is in Ansible inventory you could just use the
archive module.

From this STDOUT your root@ response only need one exit.
Because after the "exit" you get "logout" and then "Connection to
192.168.3.40 closed"

And for the sftp prompt where you did get the "No remaining responses
for" message it also only need one exit to disconnect.


--
Kai Stian Olstad