error - lxml.etree.XMLSyntaxError: xmlSAX2Characters: huge text node

[Ansel Gaddy]

I am getting an error I think may be related to how large of output I am requesting via RPC.

it works great usually I think it may be related to how large the "request support-information" is on this specific device

xml_file_rsi = dev.rpc.get_support_information()

/usr/local/share/virtualenv_eznc/lib/python2.7/site-packages/jnpr/junos/device.py:533: RuntimeWarning: An unknown exception occured - please report.
  RuntimeWarning)
Traceback (most recent call last):
  File "/usr/local/share/jtac_autoget/jtac_autoget_support.py", line 180, in <module>
    result = get_support_junos(user=u,password=p,host=r,jtac_case=jtc,filename=filename_var_logs)
  File "/usr/local/share/jtac_autoget/jtac_autoget_support.py", line 69, in get_support_junos
    xml_file_rsi = dev.rpc.get_support_information()
  File "/usr/local/share/virtualenv_eznc/lib/python2.7/site-packages/jnpr/junos/rpcmeta.py", line 156, in _exec_rpc
    return self._junos.execute(rpc, **dec_args)
  File "/usr/local/share/virtualenv_eznc/lib/python2.7/site-packages/jnpr/junos/decorators.py", line 71, in wrapper
    return function(*args, **kwargs)
  File "/usr/local/share/virtualenv_eznc/lib/python2.7/site-packages/jnpr/junos/decorators.py", line 26, in wrapper
    return function(*args, **kwargs)
  File "/usr/local/share/virtualenv_eznc/lib/python2.7/site-packages/jnpr/junos/device.py", line 516, in execute
    rpc_rsp_e = self._rpc_reply(rpc_cmd_e)
  File "/usr/local/share/virtualenv_eznc/lib/python2.7/site-packages/jnpr/junos/device.py", line 943, in _rpc_reply
    return self._conn.rpc(rpc_cmd_e)._NCElement__doc
  File "/usr/local/share/virtualenv_eznc/lib/python2.7/site-packages/ncclient/manager.py", line 172, in wrapper
    return self.execute(op_cls, *args, **kwds)
  File "/usr/local/share/virtualenv_eznc/lib/python2.7/site-packages/ncclient/manager.py", line 232, in execute
    raise_mode=self._raise_mode).request(*args, **kwds)
  File "/usr/local/share/virtualenv_eznc/lib/python2.7/site-packages/ncclient/operations/third_party/juniper/rpc.py", line 44, in request
    return self._request(rpc)
  File "/usr/local/share/virtualenv_eznc/lib/python2.7/site-packages/ncclient/operations/rpc.py", line 339, in _request
    return NCElement(self._reply, self._device_handler.transform_reply())
  File "/usr/local/share/virtualenv_eznc/lib/python2.7/site-packages/ncclient/xml_.py", line 153, in __init__
    self.__doc = self.remove_namespaces(self.__result)
  File "/usr/local/share/virtualenv_eznc/lib/python2.7/site-packages/ncclient/xml_.py", line 201, in remove_namespaces
    self.__root = etree.fromstring(str(self.__transform(etree.parse(StringIO(str(rpc_reply))))))
  File "src/lxml/lxml.etree.pyx", line 3427, in lxml.etree.parse (src/lxml/lxml.etree.c:81117)
  File "src/lxml/parser.pxi", line 1828, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:118072)
  File "src/lxml/parser.pxi", line 1848, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:118341)
  File "src/lxml/parser.pxi", line 1736, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:117021)
  File "src/lxml/parser.pxi", line 1102, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:111265)
  File "src/lxml/parser.pxi", line 595, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:105109)
  File "src/lxml/parser.pxi", line 706, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:106817)
  File "src/lxml/parser.pxi", line 635, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:105671)
  File "<string>", line 174245
lxml.etree.XMLSyntaxError: xmlSAX2Characters: huge text node, line 174245, column 1

[Ansel Gaddy]

Attached is the program.. it gathers support information and uploads it to the juniper ftp for JTAC cases.

be a little cautious running it because it will create directory based on JTAC Case and uploads files to juniper FTP 

thx,

-Ansel

[Stacy Smith]

On Friday, January 27, 2017 at 1:06:08 PM UTC-7, Ansel Gaddy wrote:

I am getting an error I think may be related to how large of output I am requesting via RPC.

it works great usually I think it may be related to how large the "request support-information" is on this specific device

xml_file_rsi = dev.rpc.get_support_information()

I believe there's a Junos bug that effects some releases where > or < characters are not properly escaped in the XML output.

Can you execute this from the CLI and see if that might be the issue you're hitting:

{master}

user@r0> request support information | display xml | match ">|<" 

<rpc-reply xmlns:junos="http://xml.juniper.net/junos/15.1F5/junos">

    <output>

    </output>

    <cli>

        <banner>{master}</banner>

    </cli>

</rpc-reply>

[Ansel Gaddy]

It doesn't appear to be the case with this issue.

user@some_router-re0> request support information | display xml | match ">|<"

<rpc-reply xmlns:junos="http://xml.juniper.net/junos/15.1F5/junos">

    <output>

    </output>

    <cli>

        <banner>{master}</banner>

    </cli>

</rpc-reply>

{master}

What leads me to believe it could be size of RPC related is that the support info is quite large here.

user@some_router-re0> request support information | display xml | count

Count: 294760 lines

{master}

with default timeout

>>> dev = Device(host='ip',user='user',password='password')

>>> dev.open(gather_facts=False)

Device(ip)

>>> xml_file_rsi = dev.rpc.get_support_information()

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/jnpr/junos/rpcmeta.py", line 156, in _exec_rpc

    return self._junos.execute(rpc, **dec_args)

  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/jnpr/junos/decorators.py", line 71, in wrapper

    return function(*args, **kwargs)

  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/jnpr/junos/decorators.py", line 26, in wrapper

    return function(*args, **kwargs)

  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/jnpr/junos/device.py", line 520, in execute

    raise EzErrors.RpcTimeoutError(self, rpc_cmd_e.tag, self.timeout)

jnpr.junos.exception.RpcTimeoutError: RpcTimeoutError(host: ip, cmd: get-support-information, timeout: 30)

with timeout increased

>>> dev.timeout = 500

>>> xml_file_rsi = dev.rpc.get_support_information()

/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/jnpr/junos/device.py:533: RuntimeWarning: An unknown exception occured - please report.

  RuntimeWarning)

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/jnpr/junos/rpcmeta.py", line 156, in _exec_rpc

    return self._junos.execute(rpc, **dec_args)

  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/jnpr/junos/decorators.py", line 71, in wrapper

    return function(*args, **kwargs)

  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/jnpr/junos/decorators.py", line 26, in wrapper

    return function(*args, **kwargs)

  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/jnpr/junos/device.py", line 516, in execute

    rpc_rsp_e = self._rpc_reply(rpc_cmd_e)

  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/jnpr/junos/device.py", line 943, in _rpc_reply

    return self._conn.rpc(rpc_cmd_e)._NCElement__doc

  File "/usr/local/lib/python2.7/site-packages/ncclient/manager.py", line 171, in wrapper

    return self.execute(op_cls, *args, **kwds)

  File "/usr/local/lib/python2.7/site-packages/ncclient/manager.py", line 231, in execute

    raise_mode=self._raise_mode).request(*args, **kwds)

  File "/usr/local/lib/python2.7/site-packages/ncclient/operations/third_party/juniper/rpc.py", line 42, in request

    return self._request(rpc)

  File "/usr/local/lib/python2.7/site-packages/ncclient/operations/rpc.py", line 339, in _request

    return NCElement(self._reply, self._device_handler.transform_reply())

  File "/usr/local/lib/python2.7/site-packages/ncclient/xml_.py", line 151, in __init__

    self.__doc = self.remove_namespaces(self.__result)

  File "/usr/local/lib/python2.7/site-packages/ncclient/xml_.py", line 199, in remove_namespaces

    self.__root = etree.fromstring(str(self.__transform(etree.parse(StringIO(str(rpc_reply))))))

  File "src/lxml/lxml.etree.pyx", line 3427, in lxml.etree.parse (src/lxml/lxml.etree.c:81110)

  File "src/lxml/parser.pxi", line 1828, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:118065)

  File "src/lxml/parser.pxi", line 1848, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:118334)

  File "src/lxml/parser.pxi", line 1736, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:117014)

  File "src/lxml/parser.pxi", line 1102, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:111258)

  File "src/lxml/parser.pxi", line 595, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:105102)

  File "src/lxml/parser.pxi", line 706, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:106810)

  File "src/lxml/parser.pxi", line 635, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:105664)

  File "<string>", line 236142

lxml.etree.XMLSyntaxError: xmlSAX2Characters: huge text node, line 236142, column 6861

Looking at the line with the issue in the RSI XML

user@router-re0> request support information | display xml | save rsi.xml

Wrote 295238 lines of output to 'rsi.xml'

{master}

user@router-re0> start shell

% sed '236142q;d' rsi.xml

        2017-02-16T19:04:59.404886+00:00 dar01 vehostd[4207]: vehostd_cmd_session_event_handler: cmd session handler called with event 0

% sed -n '236135,+20p' rsi.xml

        2017-02-16T19:04:59.403764+00:00 dar01 vehostd[4207]: shmp_server_recv_event_handler: Received header: #012#011Payload length: 0#012#011Opcode: 1#012#011Version: 1#012#011Cookie: 0

        2017-02-16T19:04:59.403769+00:00 dar01 vehostd[4207]: vehostd_get_info_cb: received GET_INFO request from 192.168.1.1:64587

        2017-02-16T19:04:59.403809+00:00 dar01 vehostd[4207]: shmp_server_send_data: Sending header (with payload): #012#011Payload length: 134#012#011Opcode: 18#012#011Version: 1#012#011Cookie: 0

        2017-02-16T19:04:59.403961+00:00 dar01 vehostd[4207]: vehostd_cmd_session_event_handler: cmd session handler called with event 1

        2017-02-16T19:04:59.403971+00:00 dar01 vehostd[4207]: handle_session_shutdown_alloc_api: Canceling pending allocs for client 17

        2017-02-16T19:04:59.403974+00:00 dar01 vehostd[4207]: vehostd_session_event_handler: Got shutdown event. Stopping session workers

        2017-02-16T19:04:59.404873+00:00 dar01 vehostd[4207]: vehostd_session_event_handler: peer 192.168.1.1:52885 connected

        2017-02-16T19:04:59.404886+00:00 dar01 vehostd[4207]: vehostd_cmd_session_event_handler: cmd session handler called with event 0

        2017-02-16T19:04:59.404890+00:00 dar01 vehostd[4207]: tcp_server_dispatch: Received 24 bytes on fd 16. Total recvd: 24

        2017-02-16T19:04:59.404893+00:00 dar01 vehostd[4207]: shmp_server_recv_event_handler: Received header: #012#011Payload length: 4#012#011Opcode: 6#012#011Version: 1#012#011Cookie: 0

        2017-02-16T19:04:59.404904+00:00 dar01 vehostd[4207]: tcp_server_dispatch: Received 4 bytes on fd 16. Total recvd: 4

        2017-02-16T19:04:59.404912+00:00 dar01 vehostd[4207]: vehostd_server_get_id_cb: received get_id {0} request from 192.168.1.1:52885

        2017-02-16T19:04:59.404937+00:00 dar01 vehostd[4207]: vehostd_client_id_get_id: Max allocated client id is 18

        2017-02-16T19:04:59.404945+00:00 dar01 vehostd[4207]: vehostd_server_get_id_cb: Allocated client id 18

        2017-02-16T19:04:59.404949+00:00 dar01 vehostd[4207]: shmp_server_send_data: Sending header (with payload): #012#011Payload length: 4#012#011Opcode: 23#012#011Version: 1#012#011Cookie: 0

        2017-02-16T19:04:59.405067+00:00 dar01 vehostd[4207]: tcp_server_dispatch: Received 24 bytes on fd 16. Total recvd: 24

        2017-02-16T19:04:59.405075+00:00 dar01 vehostd[4207]: shmp_server_recv_event_handler: Received header: #012#011Payload length: 0#012#011Opcode: 2#012#011Version: 1#012#011Cookie: 0

        2017-02-16T19:04:59.405079+00:00 dar01 vehostd[4207]: vehostd_get_processes_cb: received GET_PROCESSES {0} request from 192.168.1.1:52885

        2017-02-16T19:04:59.405083+00:00 dar01 vehostd[4207]: worker_process_create: Creating a process worker for cmd: /bin/ps -ef

        2017-02-16T19:04:59.405091+00:00 dar01 vehostd[4207]: worker_process_exec: Executing worker for cmd: /bin/ps -ef

        2017-02-16T19:04:59.423126+00:00 dar01 vehostd[4207]: init_app_server: Accepted connection. connfd = -1 errno = 4

thx

-Ansel

[Nitin Kr]

Hi Ansel,

Refer below ncclient issue for the reason

https://github.com/ncclient/ncclient/issues/185

Thanks

Nitin Kr