template "Missing end of comment tag"

[dulh...@mailbox.org]

I am trying to use template module to send 2 shell scripts to a remote.

### TASK ##################################################################################################

- name: configure postgres - template backup scripts
ansible.builtin.template:
src: "{{ item }}.j2"
dest: "{{ postgres_base }}/tools/backup/{{ item }}"
backup: yes
owner: postgres
group: postgres
mode: 0600
become: yes
loop:
- backup_postgres10.sh
- backup_archivelogs.sh
### TASK ##################################################################################################

while the first item is being templated out fine the second fails with a "Missing end of comment tag. String: #!/bin/bash" error. With 0 experience with jinja2 I can not make any sense of what I found in the documentation in that respect.
Content of both files it 100% original bash script content.


### backup_archivelogs.sh.j2 #############################################################################

#!/bin/bash
#
# $Header$
#
# backup postgres archivelogs to TSM

# source postgres env
source /opt/db/postgres/bin/.pg.env

#set -x

# write logs to backup
# do an incremental backup of the postgres archive/wal logs
# take a bunch of 20 logs per dsmc incr call
cd $PGARCH
while mapfile -t -n 20 ARCHIVE_LOGS && ((${#ARCHIVE_LOGS[@]}))
do
echo ">>> backing up: ${ARCHIVE_LOGS[@]}"
sudo /opt/tivoli/tsm/client/ba/bin/dsmc incr "${ARCHIVE_LOGS[@]}"
# check return code, if 0/true then purge logfile
RET=$?
if [ $RET -eq 0 ]; then
echo
echo ">>> TSM incremental backup successful, purging archive logs up to ${ARCHIVE_LOGS[-1]} <<<"
rm "${ARCHIVE_LOGS[@]}"
fi
sleep 2
done < <(ls -1 0*)



### OUTPUT ################################################################################################

TASK [configure postgres - template backup scripts] ********************************************************************
task path: /home/username/repos/ansible/open_source/postgres_create_service/postgres_create_service.yml:83
<server.intern> ESTABLISH SSH CONNECTION FOR USER: None
<server.intern> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o 'ControlPath="/home/username/.ansible/cp/8ca9f05e1f"' server.intern '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<server.intern> (0, b'/home/username\n', b'')
<server.intern> ESTABLISH SSH CONNECTION FOR USER: None
<server.intern> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o 'ControlPath="/home/username/.ansible/cp/8ca9f05e1f"' server.intern '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /var/tmp `"&& mkdir "` echo /var/tmp/ansible-tmp-1663162681.3683012-2675848-43258002595610 `" && echo ansible-tmp-1663162681.3683012-2675848-43258002595610="` echo /var/tmp/ansible-tmp-1663162681.3683012-2675848-43258002595610 `" ) && sleep 0'"'"''
<server.intern> (0, b'ansible-tmp-1663162681.3683012-2675848-43258002595610=/var/tmp/ansible-tmp-1663162681.3683012-2675848-43258002595610\n', b'')
Using module file /usr/lib/python3.8/site-packages/ansible/modules/stat.py
<server.intern> PUT /home/username/.ansible/tmp/ansible-local-2675679n2q7xy3g/tmpwcgv_g_c TO /var/tmp/ansible-tmp-1663162681.3683012-2675848-43258002595610/AnsiballZ_stat.py
<server.intern> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o 'ControlPath="/home/username/.ansible/cp/8ca9f05e1f"' '[server.intern]'
<server.intern> (0, b'sftp> put /home/username/.ansible/tmp/ansible-local-2675679n2q7xy3g/tmpwcgv_g_c /var/tmp/ansible-tmp-1663162681.3683012-2675848-43258002595610/AnsiballZ_stat.py\n', b'')
<server.intern> ESTABLISH SSH CONNECTION FOR USER: None
<server.intern> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o 'ControlPath="/home/username/.ansible/cp/8ca9f05e1f"' server.intern '/bin/sh -c '"'"'setfacl -m u:postgres:r-x /var/tmp/ansible-tmp-1663162681.3683012-2675848-43258002595610/ /var/tmp/ansible-tmp-1663162681.3683012-2675848-43258002595610/AnsiballZ_stat.py && sleep 0'"'"''
<server.intern> (0, b'', b'')
<server.intern> ESTABLISH SSH CONNECTION FOR USER: None
<server.intern> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o 'ControlPath="/home/username/.ansible/cp/8ca9f05e1f"' -tt server.intern '/bin/sh -c '"'"'sudo -H -S -n -u postgres /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-ztbiuccpcrwstltpesmvsnlefldjhueu ; /usr/libexec/platform-python /var/tmp/ansible-tmp-1663162681.3683012-2675848-43258002595610/AnsiballZ_stat.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<server.intern> (0, b'\r\n\r\n{"changed": false, "stat": {"exists": true, "path": "/opt/db/postgres/tools/backup/backup_postgres10.sh", "mode": "0600", "isdir": false, "ischr": false, "isblk": false, "isreg": true, "isfifo": false, "islnk": false, "issock": false, "uid": 1001, "gid": 1001, "size": 9567, "inode": 6553734, "dev": 64772, "nlink": 1, "atime": 1663162307.594887, "mtime": 1662108268.8151002, "ctime": 1662108269.4020917, "wusr": true, "rusr": true, "xusr": false, "wgrp": false, "rgrp": false, "xgrp": false, "woth": false, "roth": false, "xoth": false, "isuid": false, "isgid": false, "blocks": 24, "block_size": 4096, "device_type": 0, "readable": true, "writeable": true, "executable": false, "pw_name": "postgres", "gr_name": "postgres", "checksum": "c25840eb364bd104b796679eef8082ad6887bae3", "mimetype": "text/x-shellscript", "charset": "us-ascii", "version": "3998925608", "attributes": [], "attr_flags": ""}, "invocation": {"module_args": {"path": "/opt/db/postgres/tools/back
up/backup_postgres10.sh", "follow": false, "get_checksum": true, "checksum_algorithm": "sha1", "get_md5": false, "get_mime": true, "get_attributes": true}}}\r\n', b'Shared connection to server.intern closed.\r\n')
Using module file /usr/lib/python3.8/site-packages/ansible/modules/file.py
<server.intern> PUT /home/username/.ansible/tmp/ansible-local-2675679n2q7xy3g/tmpafre2fjl TO /var/tmp/ansible-tmp-1663162681.3683012-2675848-43258002595610/AnsiballZ_file.py
<server.intern> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o 'ControlPath="/home/username/.ansible/cp/8ca9f05e1f"' '[server.intern]'
<server.intern> (0, b'sftp> put /home/username/.ansible/tmp/ansible-local-2675679n2q7xy3g/tmpafre2fjl /var/tmp/ansible-tmp-1663162681.3683012-2675848-43258002595610/AnsiballZ_file.py\n', b'')
<server.intern> ESTABLISH SSH CONNECTION FOR USER: None
<server.intern> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o 'ControlPath="/home/username/.ansible/cp/8ca9f05e1f"' server.intern '/bin/sh -c '"'"'setfacl -m u:postgres:r-x /var/tmp/ansible-tmp-1663162681.3683012-2675848-43258002595610/ /var/tmp/ansible-tmp-1663162681.3683012-2675848-43258002595610/AnsiballZ_file.py && sleep 0'"'"''
<server.intern> (0, b'', b'')
<server.intern> ESTABLISH SSH CONNECTION FOR USER: None
<server.intern> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o 'ControlPath="/home/username/.ansible/cp/8ca9f05e1f"' -tt server.intern '/bin/sh -c '"'"'sudo -H -S -n -u postgres /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-qttckqlufxsixkcjmiiesbbvfedjlfuy ; /usr/libexec/platform-python /var/tmp/ansible-tmp-1663162681.3683012-2675848-43258002595610/AnsiballZ_file.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<server.intern> (0, b'\r\n{"path": "/opt/db/postgres/tools/backup/backup_postgres10.sh", "changed": false, "diff": {"before": {"path": "/opt/db/postgres/tools/backup/backup_postgres10.sh"}, "after": {"path": "/opt/db/postgres/tools/backup/backup_postgres10.sh"}}, "uid": 1001, "gid": 1001, "owner": "postgres", "group": "postgres", "mode": "0600", "state": "file", "secontext": "system_u:object_r:usr_t:s0", "size": 9567, "invocation": {"module_args": {"owner": "postgres", "group": "postgres", "mode": 384, "dest": "/opt/db/postgres/tools/backup/backup_postgres10.sh", "_original_basename": "backup_postgres10.sh.j2", "recurse": false, "state": "file", "path": "/opt/db/postgres/tools/backup/backup_postgres10.sh", "force": false, "follow": true, "modification_time_format": "%Y%m%d%H%M.%S", "access_time_format": "%Y%m%d%H%M.%S", "unsafe_writes": false, "_diff_peek": null, "src": null, "modification_time": null, "access_time": null, "seuser": null, "serole": null, "selevel": null, "setype": nu
ll, "attributes": null}}}\r\n', b'Shared connection to server.intern closed.\r\n')
<server.intern> ESTABLISH SSH CONNECTION FOR USER: None
<server.intern> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o 'ControlPath="/home/username/.ansible/cp/8ca9f05e1f"' server.intern '/bin/sh -c '"'"'rm -f -r /var/tmp/ansible-tmp-1663162681.3683012-2675848-43258002595610/ > /dev/null 2>&1 && sleep 0'"'"''
<server.intern> (0, b'', b'')
ok: [server.intern] => (item=backup_postgres10.sh) => changed=false
ansible_loop_var: item
checksum: c25840eb364bd104b796679eef8082ad6887bae3
dest: /opt/db/postgres/tools/backup/backup_postgres10.sh
diff:
after:
path: /opt/db/postgres/tools/backup/backup_postgres10.sh
before:
path: /opt/db/postgres/tools/backup/backup_postgres10.sh
gid: 1001
group: postgres
invocation:
module_args:
_diff_peek: null
_original_basename: backup_postgres10.sh.j2
access_time: null
access_time_format: '%Y%m%d%H%M.%S'
attributes: null
dest: /opt/db/postgres/tools/backup/backup_postgres10.sh
follow: true
force: false
group: postgres
mode: 384
modification_time: null
modification_time_format: '%Y%m%d%H%M.%S'
owner: postgres
path: /opt/db/postgres/tools/backup/backup_postgres10.sh
recurse: false
selevel: null
serole: null
setype: null
seuser: null
src: null
state: file
unsafe_writes: false
item: backup_postgres10.sh
mode: '0600'
owner: postgres
path: /opt/db/postgres/tools/backup/backup_postgres10.sh
secontext: system_u:object_r:usr_t:s0
size: 9567
state: file
uid: 1001
<server.intern> ESTABLISH SSH CONNECTION FOR USER: None
<server.intern> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o 'ControlPath="/home/username/.ansible/cp/8ca9f05e1f"' server.intern '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<server.intern> (0, b'/home/username\n', b'')
<server.intern> ESTABLISH SSH CONNECTION FOR USER: None
<server.intern> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o 'ControlPath="/home/username/.ansible/cp/8ca9f05e1f"' server.intern '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /var/tmp `"&& mkdir "` echo /var/tmp/ansible-tmp-1663162682.168117-2675848-28324059801519 `" && echo ansible-tmp-1663162682.168117-2675848-28324059801519="` echo /var/tmp/ansible-tmp-1663162682.168117-2675848-28324059801519 `" ) && sleep 0'"'"''
<server.intern> (0, b'ansible-tmp-1663162682.168117-2675848-28324059801519=/var/tmp/ansible-tmp-1663162682.168117-2675848-28324059801519\n', b'')
<server.intern> ESTABLISH SSH CONNECTION FOR USER: None
<server.intern> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o 'ControlPath="/home/username/.ansible/cp/8ca9f05e1f"' server.intern '/bin/sh -c '"'"'rm -f -r /var/tmp/ansible-tmp-1663162682.168117-2675848-28324059801519/ > /dev/null 2>&1 && sleep 0'"'"''
<server.intern> (0, b'', b'')
The full traceback is:
Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/ansible/template/__init__.py", line 1118, in do_template
t = myenv.from_string(data)
File "/usr/lib/python3.8/site-packages/jinja2/environment.py", line 880, in from_string
return cls.from_code(self, self.compile(source), globals, None)
File "/usr/lib/python3.8/site-packages/jinja2/environment.py", line 591, in compile
self.handle_exception(exc_info, source_hint=source_hint)
File "/usr/lib/python3.8/site-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.8/site-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "<unknown>", line 17, in <module>
File "/usr/lib/python3.8/site-packages/jinja2/environment.py", line 497, in _parse
return Parser(self, source, name, encode_filename(filename)).parse()
File "/usr/lib/python3.8/site-packages/jinja2/parser.py", line 901, in parse
result = nodes.Template(self.subparse(), lineno=1)
File "/usr/lib/python3.8/site-packages/jinja2/parser.py", line 872, in subparse
next(self.stream)
File "/usr/lib/python3.8/site-packages/jinja2/lexer.py", line 359, in __next__
self.current = next(self._iter)
File "/usr/lib/python3.8/site-packages/jinja2/lexer.py", line 562, in wrap
for lineno, token, value in stream:
File "/usr/lib/python3.8/site-packages/jinja2/lexer.py", line 646, in tokeniter
raise token(lineno, filename)
File "/usr/lib/python3.8/site-packages/jinja2/lexer.py", line 229, in __call__
raise self.error_class(self.message, lineno, filename)
jinja2.exceptions.TemplateSyntaxError: Missing end of comment tag

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/ansible/plugins/action/template.py", line 150, in run
resultant = templar.do_template(template_data, preserve_trailing_newlines=True, escape_backslashes=False)
File "/usr/lib/python3.8/site-packages/ansible/template/__init__.py", line 1120, in do_template
raise AnsibleError("template error while templating string: %s. String: %s" % (to_native(e), to_native(data)))
ansible.errors.AnsibleError: template error while templating string: Missing end of comment tag. String: #!/bin/bash
#
# $Header$
#
# backup postgres archivelogs to TSM

# source postgres env
source /opt/db/postgres/bin/.pg.env

#set -x

# write logs to backup
# do an incremental backup of the postgres archive/wal logs
# take a bunch of 20 logs per dsmc incr call
cd $PGARCH
while mapfile -t -n 20 ARCHIVE_LOGS && ((${#ARCHIVE_LOGS[@]}))
do
echo ">>> backing up: ${ARCHIVE_LOGS[@]}"
sudo /opt/tivoli/tsm/client/ba/bin/dsmc incr "${ARCHIVE_LOGS[@]}"
# check return code, if 0/true then purge logfile
RET=$?
if [ $RET -eq 0 ]; then
echo
echo ">>> TSM incremental backup successful, purging archive logs up to ${ARCHIVE_LOGS[-1]} <<<"
rm "${ARCHIVE_LOGS[@]}"
fi
sleep 2
done < <(ls -1 0*)


failed: [server.intern] (item=backup_archivelogs.sh) => changed=false
ansible_loop_var: item
item: backup_archivelogs.sh
msg: |-
AnsibleError: template error while templating string: Missing end of comment tag. String: #!/bin/bash
#
# $Header$
#
# backup postgres archivelogs to TSM

# source postgres env
source /opt/db/postgres/bin/.pg.env

#set -x

# write logs to backup
# do an incremental backup of the postgres archive/wal logs
# take a bunch of 20 logs per dsmc incr call
cd $PGARCH
while mapfile -t -n 20 ARCHIVE_LOGS && ((${#ARCHIVE_LOGS[@]}))
do
echo ">>> backing up: ${ARCHIVE_LOGS[@]}"
sudo /opt/tivoli/tsm/client/ba/bin/dsmc incr "${ARCHIVE_LOGS[@]}"
# check return code, if 0/true then purge logfile
RET=$?
if [ $RET -eq 0 ]; then
echo
echo ">>> TSM incremental backup successful, purging archive logs up to ${ARCHIVE_LOGS[-1]} <<<"
rm "${ARCHIVE_LOGS[@]}"
fi
sleep 2
done < <(ls -1 0*)

############################################################################################

[Todd Lewis]

The problem is, valid shell expressions like "${#ARCHIVE_LOGS[@]}" contain "{#" which is Jinja for "start-of-comment" and you have no "#}" end-of-comment.

To get the string "${#ARCHIVE_LOGS[@]}" to be generated in a template, you have to split the "{" from the "#". Here's one way:

    while mapfile -t -n 20 ARCHIVE_LOGS && ((${{ '{' ~ '#' }}ARCHIVE_LOGS[@]}))

There are lots of other ways to slice it up though. Good luck.

[dulh...@mailbox.org]

sorry false alarm. my backup_archivelogs.sh file was completely off

> --
> You received this message because you are subscribed to the Google Groups "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/1421900087.383940.1663164196876%40office.mailbox.org.