[error] lxml.etree.XMLSyntaxError while 'Updating CVE's...'

Denis Kostromitskiy

unread,

Jul 9, 2014, 11:05:39 AM7/9/14

to vfe...@googlegroups.com

Hi guys!
Can you help me with following error I get while running: sudo python tp/src/scripts/initialize_vFense.py..
...
Updating CVE's...
Traceback (most recent call last):
File "tp/src/scripts/initialize_vFense.py", line 340, in <module>
    db_initialized, msg = initialize_db()
File "tp/src/scripts/initialize_vFense.py", line 283, in initialize_db
    load_up_all_xml_into_db()
File "/usr/local/lib/python2.7/dist-packages/vFense/plugins/vuln/cve/parser.py", line 366, in load_up_all_xml_into_db
    parse_cve_and_udpatedb(False, nvd_files)
File "/usr/local/lib/python2.7/dist-packages/vFense/plugins/vuln/cve/parser.py", line 314, in parse_cve_and_udpatedb
    for event, entry in etree.iterparse(nvd_file, events=['start', 'end']):
File "iterparse.pxi", line 478, in lxml.etree.iterparse.__next__ (src/lxml/lxml.etree.c:98432)
File "iterparse.pxi", line 530, in lxml.etree.iterparse._read_more_events (src/lxml/lxml.etree.c:98953)
File "parser.pxi", line 590, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:74696)
lxml.etree.XMLSyntaxError: Extra content at the end of the document, line 203050, column 60

Denis Kostromitskiy

unread,

Jul 9, 2014, 1:39:09 PM7/9/14

to vfe...@googlegroups.com

I'm running Ubuntu 12.04 32bit.

Allen Sanabria

unread,

Jul 9, 2014, 1:56:18 PM7/9/14

to vfe...@googlegroups.com

I've never tried this on a 32bit server, I would go ahead and try on a 64 bit. Also from the error, it looks like the file did not download correctly, which caused that error. Please try the install again, but this time on a 64bit system with at least 6G of ram.

Denis Kostromitskiy

unread,

Jul 10, 2014, 7:24:42 AM7/10/14

to vfe...@googlegroups.com

Hi! For some reason it went fine today. And db is filled out.
But now I'm getting next error while running

sudo /opt/TopPatch/tp/src/daemon/vFensed start

patching Server is starting. Please wait....
RqWorker Queue incoming_updates started successfully

RqWorker Queue downloader started successfully

RqWorker Queue stats started successfully

RqWorker Queue updater started successfully

RqWorker Queue post_store_operation started successfully

RqWorker Queue agent_status started successfully

RqWorker Queue create_secondary_indexes started successfully

RqWorker Queue delete_agent started successfully

RqWorker Queue move_agent started successfully

patching Server has been started. Enjoy !!!
denis@denis-VirtualBox:/usr/local/lib/python2.7/dist-packages$ Traceback (most recent call last):
 File "src/scheduler/rvscheduler.py", line 5, in <module>
 from vFense.scheduler.jobManager import start_scheduler, job_exists, remove_job
 File "/usr/local/lib/python2.7/dist-packages/vFense/scheduler/jobManager.py", line 11, in <module>
 from apscheduler.jobstores.redis_store import RedisJobStore
 File "/usr/local/lib/python2.7/dist-packages/apscheduler/jobstores/redis_store.py", line 19, in <module>
 raise ImportError('RedisJobStore requires redis installed')
ImportError: RedisJobStore requires redis installed
14:49:52 RQ worker started, version 0.4.6
14:49:52 RQ worker started, version 0.4.6
14:49:52 
14:49:52 *** Listening on incoming_updates...
14:49:52 
14:49:53 *** Listening on stats...
14:49:53 RQ worker started, version 0.4.6
14:49:53 
14:49:53 *** Listening on downloader...
14:49:53 RQ worker started, version 0.4.6
14:49:53 
14:49:53 *** Listening on updater...
14:49:53 RQ worker started, version 0.4.6
14:49:53 RQ worker started, version 0.4.6
14:49:53 
14:49:53 *** Listening on create_secondary_indexes...
14:49:53 
14:49:53 *** Listening on post_store_operation...
14:49:53 RQ worker started, version 0.4.6
14:49:53 RQ worker started, version 0.4.6
14:49:53 
14:49:53 *** Listening on delete_agent...
14:49:53 
14:49:53 *** Listening on agent_status...
14:49:53 RQ worker started, version 0.4.6
14:49:53 
14:49:53 *** Listening on move_agent...


Traceback (most recent call last):
Traceback (most recent call last):


 File "src/vFense_listener.py", line 18, in <module>
 import vFense_module_loader
 File "/opt/TopPatch/tp/src/vFense_module_loader.py", line 31, in <module>
 from vFense.server.api.scheduler_api import (ScheduleListerHandler,
 File "/usr/local/lib/python2.7/dist-packages/vFense/server/api/scheduler_api.py", line 10, in <module>
 from vFense.scheduler.jobManager import *
 File "/usr/local/lib/python2.7/dist-packages/vFense/scheduler/jobManager.py", line 11, in <module>
 from apscheduler.jobstores.redis_store import RedisJobStore
 File "/usr/local/lib/python2.7/dist-packages/apscheduler/jobstores/redis_store.py", line 19, in <module>
 raise ImportError('RedisJobStore requires redis installed')
ImportError: RedisJobStore requires redis installed
 File "src/vFense_listener.py", line 18, in <module>
 import vFense_module_loader
 File "/opt/TopPatch/tp/src/vFense_module_loader.py", line 31, in <module>
 from vFense.server.api.scheduler_api import (ScheduleListerHandler,
 File "/usr/local/lib/python2.7/dist-packages/vFense/server/api/scheduler_api.py", line 10, in <module>
 from vFense.scheduler.jobManager import *
 File "/usr/local/lib/python2.7/dist-packages/vFense/scheduler/jobManager.py", line 11, in <module>
 from apscheduler.jobstores.redis_store import RedisJobStore
 File "/usr/local/lib/python2.7/dist-packages/apscheduler/jobstores/redis_store.py", line 19, in <module>
 raise ImportError('RedisJobStore requires redis installed')
ImportError: RedisJobStore requires redis installed

Traceback (most recent call last):


 File "src/vFense_listener.py", line 18, in <module>
 import vFense_module_loader
 File "/opt/TopPatch/tp/src/vFense_module_loader.py", line 31, in <module>
 from vFense.server.api.scheduler_api import (ScheduleListerHandler,
 File "/usr/local/lib/python2.7/dist-packages/vFense/server/api/scheduler_api.py", line 10, in <module>


 Traceback (most recent call last):


 File "src/vFense_listener.py", line 18, in <module>
 import vFense_module_loader
 File "/opt/TopPatch/tp/src/vFense_module_loader.py", line 31, in <module>
 from vFense.server.api.scheduler_api import (ScheduleListerHandler,
 File "/usr/local/lib/python2.7/dist-packages/vFense/server/api/scheduler_api.py", line 10, in <module>
 from vFense.scheduler.jobManager import *
 File "/usr/local/lib/python2.7/dist-packages/vFense/scheduler/jobManager.py", line 11, in <module>
 from apscheduler.jobstores.redis_store import RedisJobStore
 File "/usr/local/lib/python2.7/dist-packages/apscheduler/jobstores/redis_store.py", line 19, in <module>
 raise ImportError('RedisJobStore requires redis installed')
ImportError: RedisJobStore requires redis installed

Traceback (most recent call last):

from vFense.scheduler.jobManager import *
 File "/usr/local/lib/python2.7/dist-packages/vFense/scheduler/jobManager.py", line 11, in <module>
 from apscheduler.jobstores.redis_store import RedisJobStore
 File "/usr/local/lib/python2.7/dist-packages/apscheduler/jobstores/redis_store.py", line 19, in <module>
 raise ImportError('RedisJobStore requires redis installed')
ImportError: RedisJobStore requires redis installed
 File "src/vFense_listener.py", line 18, in <module>
 import vFense_module_loader
 File "/opt/TopPatch/tp/src/vFense_module_loader.py", line 31, in <module>
 from vFense.server.api.scheduler_api import (ScheduleListerHandler,
 File "/usr/local/lib/python2.7/dist-packages/vFense/server/api/scheduler_api.py", line 10, in <module>
 from vFense.scheduler.jobManager import *
 File "/usr/local/lib/python2.7/dist-packages/vFense/scheduler/jobManager.py", line 11, in <module>
 from apscheduler.jobstores.redis_store import RedisJobStore
 File "/usr/local/lib/python2.7/dist-packages/apscheduler/jobstores/redis_store.py", line 19, in <module>
 raise ImportError('RedisJobStore requires redis installed')
ImportError: RedisJobStore requires redis installed

Traceback (most recent call last):


 File "src/vFense_listener.py", line 18, in <module>
 import vFense_module_loader
 File "/opt/TopPatch/tp/src/vFense_module_loader.py", line 31, in <module>
 from vFense.server.api.scheduler_api import (ScheduleListerHandler,
 File "/usr/local/lib/python2.7/dist-packages/vFense/server/api/scheduler_api.py", line 10, in <module>
 from vFense.scheduler.jobManager import *
 File "/usr/local/lib/python2.7/dist-packages/vFense/scheduler/jobManager.py", line 11, in <module>
 from apscheduler.jobstores.redis_store import RedisJobStore
 File "/usr/local/lib/python2.7/dist-packages/apscheduler/jobstores/redis_store.py", line 19, in <module>
 raise ImportError('RedisJobStore requires redis installed')
ImportError: RedisJobStore requires redis installed

Traceback (most recent call last):


 File "src/vFense_listener.py", line 18, in <module>
 import vFense_module_loader

Traceback (most recent call last):


 File "src/vFense_web.py", line 19, in <module>
 import vFense_module_loader
 File "/opt/TopPatch/tp/src/vFense_module_loader.py", line 31, in <module>
 from vFense.server.api.scheduler_api import (ScheduleListerHandler,
 File "/usr/local/lib/python2.7/dist-packages/vFense/server/api/scheduler_api.py", line 10, in <module>
 from vFense.scheduler.jobManager import *
 File "/usr/local/lib/python2.7/dist-packages/vFense/scheduler/jobManager.py", line 11, in <module>
 File "/opt/TopPatch/tp/src/vFense_module_loader.py", line 31, in <module>
from apscheduler.jobstores.redis_store import RedisJobStore
 File "/usr/local/lib/python2.7/dist-packages/apscheduler/jobstores/redis_store.py", line 19, in <module>
 from vFense.server.api.scheduler_api import (ScheduleListerHandler,
 File "/usr/local/lib/python2.7/dist-packages/vFense/server/api/scheduler_api.py", line 10, in <module>
 raise ImportError('RedisJobStore requires redis installed')
ImportErrorfrom vFense.scheduler.jobManager import *
 File "/usr/local/lib/python2.7/dist-packages/vFense/scheduler/jobManager.py", line 11, in <module>
 : RedisJobStore requires redis installed
from apscheduler.jobstores.redis_store import RedisJobStore
 File "/usr/local/lib/python2.7/dist-packages/apscheduler/jobstores/redis_store.py", line 19, in <module>
 raise ImportError('RedisJobStore requires redis installed')
ImportError: RedisJobStore requires redis installed

Traceback (most recent call last):


 File "src/vFense_listener.py", line 18, in <module>
 import vFense_module_loader
 File "/opt/TopPatch/tp/src/vFense_module_loader.py", line 31, in <module>
 from vFense.server.api.scheduler_api import (ScheduleListerHandler,
 File "/usr/local/lib/python2.7/dist-packages/vFense/server/api/scheduler_api.py", line 10, in <module>
 from vFense.scheduler.jobManager import *
 File "/usr/local/lib/python2.7/dist-packages/vFense/scheduler/jobManager.py", line 11, in <module>
 from apscheduler.jobstores.redis_store import RedisJobStore
 File "/usr/local/lib/python2.7/dist-packages/apscheduler/jobstores/redis_store.py", line 19, in <module>
 raise ImportError('RedisJobStore requires redis installed')
ImportError: RedisJobStore requires redis installed

Traceback (most recent call last):
Traceback (most recent call last):


 File "src/vFense_listener.py", line 18, in <module>
 import vFense_module_loader
 File "/opt/TopPatch/tp/src/vFense_module_loader.py", line 31, in <module>
 from vFense.server.api.scheduler_api import (ScheduleListerHandler,
 File "/usr/local/lib/python2.7/dist-packages/vFense/server/api/scheduler_api.py", line 10, in <module>
 from vFense.scheduler.jobManager import *
 File "/usr/local/lib/python2.7/dist-packages/vFense/scheduler/jobManager.py", line 11, in <module>
 from apscheduler.jobstores.redis_store import RedisJobStore
 File "/usr/local/lib/python2.7/dist-packages/apscheduler/jobstores/redis_store.py", line 19, in <module>
 File "src/vFense_listener.py", line 18, in <module>
 raise ImportError('RedisJobStore requires redis installed')
ImportError : RedisJobStore requires redis installed
import vFense_module_loader
 File "/opt/TopPatch/tp/src/vFense_module_loader.py", line 31, in <module>
 from vFense.server.api.scheduler_api import (ScheduleListerHandler,
 File "/usr/local/lib/python2.7/dist-packages/vFense/server/api/scheduler_api.py", line 10, in <module>
 from vFense.scheduler.jobManager import *
 File "/usr/local/lib/python2.7/dist-packages/vFense/scheduler/jobManager.py", line 11, in <module>
 from apscheduler.jobstores.redis_store import RedisJobStore
 File "/usr/local/lib/python2.7/dist-packages/apscheduler/jobstores/redis_store.py", line 19, in <module>
 raise ImportError('RedisJobStore requires redis installed')
ImportError: RedisJobStore requires redis installed

but  I have  redis  installed.

running just one file gives the same error

python /usr/local/lib/python2.7/dist-packages/apscheduler/jobstores/redis_store.py

Traceback (most recent call last):


 File "/usr/local/lib/python2.7/dist-packages/apscheduler/jobstores/redis_store.py", line 19, in <module>
 raise ImportError('RedisJobStore requires redis installed')
ImportError: RedisJobStore requires redis installed

среда, 9 июля 2014 г., 21:56:18 UTC+4 пользователь Allen Sanabria написал:

Allen Sanabria

unread,

Jul 10, 2014, 7:25:50 AM7/10/14

to Denis Kostromitskiy, vfe...@googlegroups.com

Redis does need to be installed. sudo apt-get install redis and sudo pip install pyredis

--
You received this message because you are subscribed to a topic in the Google Groups "vFense" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vfense/ZkBWlGzac2w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vfense+un...@googlegroups.com.
To post to this group, send email to vfe...@googlegroups.com.
Visit this group at http://groups.google.com/group/vfense.
To view this discussion on the web visit https://groups.google.com/d/msgid/vfense/d6940437-1f19-4283-9410-4860c9ce46cb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Denis Kostromitskiy

unread,

Jul 10, 2014, 7:29:54 AM7/10/14

to vfe...@googlegroups.com, mojo...@gmail.com

Redis there
sudo service redis-server status
redis-server is running

and
sudo pip install redis
Requirement already satisfied (use --upgrade to upgrade): redis in /usr/local/lib/python2.7/dist-packages

for pyredis
sudo pip install pyredis
Downloading/unpacking pyredis
Could not find any downloads that satisfy the requirement pyredis
No distributions at all found for pyredis

четверг, 10 июля 2014 г., 15:25:50 UTC+4 пользователь Allen Sanabria написал:

sudo /opt/TopPatch/tp/src/daemon/vFensed start



 File "src/vFense_listener.py", line 18<span sty

...

Allen Sanabria

unread,

Jul 10, 2014, 7:34:51 AM7/10/14

to Denis Kostromitskiy, vfe...@googlegroups.com

sudo pip uninstall apscheduler

sudo pip install apscheduler==2.1.2

--

You received this message because you are subscribed to a topic in the Google Groups "vFense" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vfense/ZkBWlGzac2w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vfense+un...@googlegroups.com.
To post to this group, send email to vfe...@googlegroups.com.
Visit this group at http://groups.google.com/group/vfense.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfense/565c8e1c-10b1-48f6-8d7c-ee2528e478e2%40googlegroups.com.

Denis Kostromitskiy

unread,

Jul 10, 2014, 9:28:50 AM7/10/14

to vfe...@googlegroups.com, mojo...@gmail.com

Unfortunately this does not help (

четверг, 10 июля 2014 г., 15:34:51 UTC+4 пользователь Allen Sanabria написал:

Allen Sanabria

unread,

Jul 10, 2014, 12:35:35 PM7/10/14

to vfe...@googlegroups.com, mojo...@gmail.com

I started 2 fresh vms with Ubuntu 14.04 and and Ubuntu 12.04

vfense@vFense:~$ sudo ln -s /opt/TopPatch/tp/src /usr/local/lib/python2.7/dist-packages/vFense

vfense@vFense:~$ sudo chown -R vfense.vfense /opt/TopPatch
vfense@vFense:~$ mkdir -p /opt/TopPatch/var/log
vfense@vFense:~$ sudo python /opt/TopPatch/tp/src/scripts/initialize_vFense.py --password=vFense#123
starting listener
listener started
Im about to connect to the listener
localhost 9004
I'm connected to the listener
I'm sending the new config file over
new config sent
shutting down listener
listener is down
Rethink is not Running
Rethink instances.d directory removed and cleaned
update-rc.d: /etc/init.d/vFense: file does not exist
patching file /usr/local/lib/python2.7/dist-packages/apscheduler/scheduler.py
Sleeping until rethink starts
rethinkdb: vFense: Starting instance. (logging to `/var/lib/rethinkdb/vFense/data/log_file')
Admin username = admin
Admin password = vFense#123
Agent api user = agent_api
Agent password = >0M=eI?N
Updating CVE's...
Done Updating CVE's...
Updating Microsoft Security Bulletin Ids...
Done Updating Microsoft Security Bulletin Ids...
Updating Ubuntu Security Bulletin Ids...( This can take a couple of minutes )
Done Updating Ubuntu Security Bulletin Ids...
vFense environment has been succesfully initialized

Denis Kostromitskiy

unread,

Jul 10, 2014, 1:40:59 PM7/10/14

to Allen Sanabria, vfe...@googlegroups.com

Thanks for info,Allen.But as I wrote db filled out correctly.Struggles I'm facing comes from
sudo /opt/TopPatch/tp/src/daemon/vFensed start

Will try to rebuild all from scratch and let you know.

Allen Sanabria

unread,

Jul 10, 2014, 2:02:04 PM7/10/14

to Denis Kostromitskiy, vfe...@googlegroups.com

My apologies, I forgot to include the start output.

vfense@vFense:~$ python /opt/TopPatch/tp/src/daemon/vFensed start

patching Server is starting. Please wait....

RqWorker Queue incoming_updates started successfully

RqWorker Queue downloader started successfully

RqWorker Queue stats started successfully

RqWorker Queue updater started successfully

RqWorker Queue post_store_operation started successfully

RqWorker Queue agent_status started successfully

RqWorker Queue create_secondary_indexes started successfully

RqWorker Queue delete_agent started successfully

RqWorker Queue move_agent started successfully