How to discriminate between chunked and not chunked answers from uri

[jean-christophe manciot]

ansible core 2.12.3

I am unable to discriminate with uri between responses which have been chunked and those which haven't.

In both cases, I get the same:

- msg: OK (unknown bytes)

- transfer_encoding: chunked

This happens for instance when using uri over CSR 1kv 17.3.1a with:

- name: Sending GET datastores with uri
  ansible.builtin.uri:
        body: ''
        body_format: 'json'
        force_basic_auth: yes
        headers:
          Accept: application/yang-data+json
          Content-Type: application/yang-data+json
        method: GET
        return_content: yes
        status_code: "100,101,102,200,201,202,203,204,205,206,207,208,226,300,301,302,303,304,305,307,308"
        timeout: 120
        url: "https://172.21.126.182/restconf/data/ietf-netconf-monitoring:netconf-state/datastores?content=nonconfig"
        url_password: password
        url_username: admin
        validate_certs: false
  register: return_uri_restconf_json

leads to:

task path: main.yml:219
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/bash -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/bash -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1646997734.4119265-2691638-25534722658108 `" && echo ansible-tmp-1646997734.4119265-2691638-25534722658108="` echo /root/.ansible/tmp/ansible-tmp-1646997734.4119265-2691638-25534722658108 `" ) && sleep 0'
Using module file /usr/local/lib/python3.9/dist-packages/ansible/modules/uri.py
<localhost> PUT /root/.ansible/tmp/ansible-local-269044374_5i0rc/tmpyvffo2y1 TO /root/.ansible/tmp/ansible-tmp-1646997734.4119265-2691638-25534722658108/AnsiballZ_uri.py
<localhost> EXEC /bin/bash -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1646997734.4119265-2691638-25534722658108/ /root/.ansible/tmp/ansible-tmp-1646997734.4119265-2691638-25534722658108/AnsiballZ_uri.py && sleep 0'
<localhost> EXEC /bin/bash -c '/usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1646997734.4119265-2691638-25534722658108/AnsiballZ_uri.py && sleep 0'
<localhost> EXEC /bin/bash -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1646997734.4119265-2691638-25534722658108/ > /dev/null 2>&1 && sleep 0'
ok: [CSR1000v-17.3.1a -> localhost] => changed=false
  cache_control: private, no-cache, must-revalidate, proxy-revalidate
  connection: close
  content: |-
    {
      "ietf-netconf-monitoring:datastores": {
        "datastore": [
          {
            "name": "running",
            "tailf-netconf-monitoring:transaction-id": "1646-983921-976163"
          },
          {
            "name": "candidate"
          }
        ]
      }
    }
  content_type: application/yang-data+json
  cookies: {}
  cookies_string: ''
  date: Fri, 11 Mar 2022 11:22:14 GMT
  elapsed: 0
  invocation:
    module_args:
      attributes: null
      body: ''
      body_format: json
      ca_path: null
      client_cert: null
      client_key: null
      creates: null
      dest: null
      follow_redirects: safe
      force: false
      force_basic_auth: true
      group: null
      headers:
        Accept: application/yang-data+json
        Content-Type: application/yang-data+json
      http_agent: ansible-httpget
      method: GET
      mode: null
      owner: null
      remote_src: false
      removes: null
      return_content: true
      selevel: null
      serole: null
      setype: null
      seuser: null
      src: null
      status_code:
      - 100
      - 101
      - 102
      - 200
      - 201
      - 202
      - 203
      - 204
      - 205
      - 206
      - 207
      - 208
      - 226
      - 300
      - 301
      - 302
      - 303
      - 304
      - 305
      - 307
      - 308
      timeout: 120
      unix_socket: null
      unredirected_headers: []
      unsafe_writes: false
      url: https://172.21.126.182/restconf/data/ietf-netconf-monitoring:netconf-state/datastores?content=nonconfig
      url_password: VALUE_SPECIFIED_IN_NO_LOG_PARAMETER
      url_username: admin
      use_gssapi: false
      use_proxy: true
      validate_certs: false
  json:
    ietf-netconf-monitoring:datastores:
      datastore:
      - name: running
        tailf-netconf-monitoring:transaction-id: 1646-983921-976163
      - name: candidate
  msg: OK (unknown bytes)
  pragma: no-cache
  redirected: false
  server: openresty
  status: 200
  transfer_encoding: chunked
  url: https://172.21.126.182/restconf/data/ietf-netconf-monitoring:netconf-state/datastores?content=nonconfig

We can see that the response has NOT been chunked.

In another example, the response has been chunked with:

- name: Sending GET netconf-state with uri
  ansible.builtin.uri:
        body: ''
        body_format: 'json'
        force_basic_auth: yes
        headers:
          Accept: application/yang-data+json
          Content-Type: application/yang-data+json
        method: GET
        return_content: yes
        status_code: "100,101,102,200,201,202,203,204,205,206,207,208,226,300,301,302,303,304,305,307,308"
        timeout: 120
        url: "https://172.21.126.182/restconf/data/ietf-netconf-monitoring:netconf-state?content=nonconfig"
        url_password: password
        url_username: admin
        validate_certs: false
  register: return_uri_restconf_json

which leads to a really chunked answer:

task path: main.yml:219
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/bash -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/bash -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1646997777.3538132-2700276-93994275105673 `" && echo ansible-tmp-1646997777.3538132-2700276-93994275105673="` echo /root/.ansible/tmp/ansible-tmp-1646997777.3538132-2700276-93994275105673 `" ) && sleep 0'
Using module file /usr/local/lib/python3.9/dist-packages/ansible/modules/uri.py
<localhost> PUT /root/.ansible/tmp/ansible-local-269044374_5i0rc/tmps9ux33ea TO /root/.ansible/tmp/ansible-tmp-1646997777.3538132-2700276-93994275105673/AnsiballZ_uri.py
<localhost> EXEC /bin/bash -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1646997777.3538132-2700276-93994275105673/ /root/.ansible/tmp/ansible-tmp-1646997777.3538132-2700276-93994275105673/AnsiballZ_uri.py && sleep 0'
<localhost> EXEC /bin/bash -c '/usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1646997777.3538132-2700276-93994275105673/AnsiballZ_uri.py && sleep 0'
<localhost> EXEC /bin/bash -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1646997777.3538132-2700276-93994275105673/ > /dev/null 2>&1 && sleep 0'
ok: [CSR1000v-17.3.1a -> localhost] => changed=false
  cache_control: private, no-cache, must-revalidate, proxy-revalidate
  connection: close
  content: |-
    {
      "ietf-netconf-monitoring:netconf-state": {
        "capabilities": {
...
        },
        "datastores": {
          "datastore": [
            {
              "name": "running",
              "tailf-netconf-monitoring:transaction-id": "1646-983921-976163"
            },
            {
              "name": "candidate"
            }
          ]
        },
        "schemas": {
...
        },
        "sessions": {
          "session": [
            {
              "session-id": 26,
              "transport": "tailf-netconf-monitoring:rest-http",
              "username": "admin",
              "source-host": "172.21.0.1",
              "login-time": "2022-03-11T11:22:57+00:00",
              "tailf-netconf-monitoring:transaction": [
                {
  content_type: application/yang-data+json
  cookies: {}
  cookies_string: ''
  date: Fri, 11 Mar 2022 11:22:57 GMT
  elapsed: 0
  invocation:
    module_args:
      attributes: null
      body: ''
      body_format: json
      ca_path: null
      client_cert: null
      client_key: null
      creates: null
      dest: null
      follow_redirects: safe
      force: false
      force_basic_auth: true
      group: null
      headers:
        Accept: application/yang-data+json
        Content-Type: application/yang-data+json
      http_agent: ansible-httpget
      method: GET
      mode: null
      owner: null
      remote_src: false
      removes: null
      return_content: true
      selevel: null
      serole: null
      setype: null
      seuser: null
      src: null
      status_code:
      - 100
      - 101
      - 102
      - 200
      - 201
      - 202
      - 203
      - 204
      - 205
      - 206
      - 207
      - 208
      - 226
      - 300
      - 301
      - 302
      - 303
      - 304
      - 305
      - 307
      - 308
      timeout: 120
      unix_socket: null
      unredirected_headers: []
      unsafe_writes: false
      url: https://172.21.126.182/restconf/data/ietf-netconf-monitoring:netconf-state?content=nonconfig
      url_password: VALUE_SPECIFIED_IN_NO_LOG_PARAMETER
      url_username: admin
      use_gssapi: false
      use_proxy: true
      validate_certs: false
  msg: OK (unknown bytes)
  pragma: no-cache
  redirected: false
  server: openresty
  status: 200
  transfer_encoding: chunked
  url: https://172.21.126.182/restconf/data/ietf-netconf-monitoring:netconf-state?content=nonconfig

Any suggestion about how to differentiate between both types of responses?

[Matt Martz]

Python 3 should be attempting to read all chunks if the transfer-encoding is chunked when a single `.read()` is called.  I would not expect it to fail to fetch all chunks.  As such, even if `transfer_encoding: chunked` is displayed, python should have still fetched the full content.

So, to more specifically answer your question, if you see `transfer_encoding: chunked`, then the response was chunked, but python has attempted to fetch all chunks.

I suppose to determine if you have a partially read chunk, and not the full response, you would have to look at whether the `.json` key exists on the module response.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/9cc40b9d-20c2-45f6-a031-29dadef118c8n%40googlegroups.com.

--

Matt Martz
@sivel
sivel.net

[jean-christophe manciot]

Thanks.

But I understand from your answer that really chunked answers should not happen under normal circumstances.

Since "Python 3 should be attempting to read all chunks", I tried to set the timeout to a high value (3600) to eliminate that potential explanation for the chunked answer.

However, it is still happening.

Should I report this a bug or can you suggest anything else?

Also, by default (if no other headers are set), is it possible to receive something else than 'transfer_encoding: chunked' from the server for body_format: 'json' or body_format: 'raw'?

[Matt Martz]

I wouldn't know where an appropriate place to file a bug would be.  Looking at the CPython code for http.client, `.read()` should be fully consuming all of the chunks.

https://github.com/python/cpython/blob/3.10/Lib/http/client.py#L577

As such, I doubt we would do anything in ansible-core to deal with this.

There is no way for the client to instruct the server to not send a chunked response.  A chunked response is used when the Content-Length isn't known at the time the response is sent back to the client.

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/c9fb96b2-124e-4d88-a65f-3e2f5dfdd2e4n%40googlegroups.com.

[jean-christophe manciot]

But how can the Content-Length be unknown at the time the response is sent back to the client for small unchunked answers (case n°1 of my first post)?

[jean-christophe manciot]

I have filed the issue here