IOError: File not open for reading

[Kent Coble]

I have an error that is thrown part-way into a script we are using. The script runs every 5 minutes (in a self-contained loop) to SNMP anywhere from 300 to 2,700 devices and upload it to an ElasticSearch server.

Calling @memprof seems to cause issues at the first function that finishes. The stack trace for the error is as follows:

Fetch Number of Stations per IAP

================================

memprof starting (min. size: 1048576)

************

0s (+0s)

************

Checking alive hosts...

Ready.

Running. . .

memprof starting (min. size: 1048576)

************

0s (+0s)

************

************

12s (+12s)

************

Traceback (most recent call last):

  File "./getStationsPerIAP.py", line 281, in <module>

    f = Front().init(arrayList=args.file, verbose=args.verbose, sleep=args.sleep, reset=args.reset) #desiredOID=args.OID,

  File "./getStationsPerIAP.py", line 36, in init

    self.Start()

  File "/usr/local/lib/python2.7/dist-packages/memprof/memprof.py", line 151, in __call__

    res = self.func(*args, **kwargs)

  File "./getStationsPerIAP.py", line 103, in Start

    self.Query(expand, 'expand', var)

  File "/usr/local/lib/python2.7/dist-packages/memprof/memprof.py", line 151, in __call__

    res = self.func(*args, **kwargs)

  File "./getStationsPerIAP.py", line 227, in Query

    results = session.seqwalk()

  File "/usr/local/lib/python2.7/dist-packages/fastsnmpy.py", line 45, in seqwalk

    Community = self.community

  File "build/bdist.linux-x86_64/egg/netsnmp/client.py", line 119, in __init__

  File "build/bdist.linux-x86_64/egg/netsnmp/client.py", line 119, in __init__

  File "/usr/local/lib/python2.7/dist-packages/memprof/memprof.py", line 88, in tracer

    self.checkMem()

  File "/usr/local/lib/python2.7/dist-packages/memprof/memprof.py", line 108, in checkMem

    size = getSize(value)

  File "memprof/getsize.pyx", line 53, in memprof.getsize.getSize (memprof/getsize.c:1516)

  File "memprof/getsize.pyx", line 48, in memprof.getsize.getSize.size_of (memprof/getsize.c:1328)

IOError: File not open for reading

(I've included the original script should it help)

_________

Ubuntu 14.10 (Server)

Python 2.7.8

Account has admin rights (e.g. can use sudo for installing packages)

All packages are up-to-update via Pip

If there's anything I can do please let me know

[J.M. Dana]

Hello Kent,

Thanks a lot for your feedback.

The problem seems to be related with the differences between Python 2
and Python 3 file objects.

I have just released a new version in both github and Pypi that you can
install using pip (just make sure that you use the --upgrade flag).

Please let me know if that solves your problem.

Regards,
Jose.

> *File "/usr/local/lib/python2.7/dist-packages/memprof/memprof.py", line
> 88, in tracer*
> * self.checkMem()*
> * File "/usr/local/lib/python2.7/dist-packages/memprof/memprof.py", line
> 108, in checkMem*
> * size = getSize(value)*
> * File "memprof/getsize.pyx", line 53, in memprof.getsize.getSize
> (memprof/getsize.c:1516)*
> * File "memprof/getsize.pyx", line 48, in memprof.getsize.getSize.size_of
> (memprof/getsize.c:1328)*
> *IOError: File not open for reading*

>
>
> (I've included the original script should it help)
>
>
> _________
>
> Ubuntu 14.10 (Server)
> Python 2.7.8
> Account has admin rights (e.g. can use sudo for installing packages)
> All packages are up-to-update via Pip
>
>
>
> If there's anything I can do please let me know
>

> --
> http://jmdana.github.io/memprof/
> http://github.com/jmdana/memprof
> ---
> You received this message because you are subscribed to the Google Groups "memprof" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to memprof+u...@googlegroups.com.
> To post to this group, send email to mem...@googlegroups.com.
> Visit this group at http://groups.google.com/group/memprof.
> To view this discussion on the web visit https://groups.google.com/d/msgid/memprof/6542a2ee-0e2f-4f66-b394-84a7e0a9c6ca%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

> #!/usr/bin/env python
>
> """
> *config file for timing of script
> *auto update array list
> multiple or one mega script?
>
> """
>
> from __future__ import print_function
> import sys, os, argparse, gc
> from fastsnmpy import SnmpSession
> import time, datetime
> import stopwatch
> from socket import gethostbyname #,gethostbyaddr, in contrast, would get the hostname from the IPAddress
> import json, requests
>
> from memprof import * # Memory debugging/tracing
>
> try:
> input = raw_input
> except NameError:
> pass
>
>
> class Front():
> def init(self, arrayList, desiredOID = ('arrayModel','iapNumStations','iapChannel'), verbose=0, sleep=120, reset=180):
> print('Fetch Number of Stations per IAP')
> print('================================')
>
> self.arrayList = arrayList
> self.sleep = sleep
> self.verbose = verbose
> self.desiredOID = desiredOID
> self.reset = reset * 60 # Convert to seconds, else we'll lose timer values if loop finishes too quickly
> self.Start()
>
> housing = ('HR0','HR1','HR2','HR3','HL','WP','FLSR','W0');
>
> @memprof(plot = True)
> def Start(self):
> #debug = int(input('What level of debugging do you want? (Minimal = 0, Verbose = 1): '))
> #seconds = int(input('Desired time between querying the list (in seconds): '))
> modified = self.LastModified(self.arrayList) # Initialize variables
> f = open(self.arrayList, 'r')
> expand = []
> expanded = []
> extend = []
> print ('Checking alive hosts...')
> for line in f:
> line = line.strip()
> pingres = os.system('ping -c 1 -W1 -t 255 ' + line + ' > /dev/null 2>&1')
> if pingres == 0 : # ifalive
> if line.startswith(self.housing):
> extend.append(line)
> elif (line[0].isdigit() or line[0].lower() <= 'i'):
> expand.append(line)
> else:
> expanded.append(line)
>
> f.close()
>
> #var = [ self.desiredOID ]
> var = list(self.desiredOID)
>
> print ('Ready.')
> reset = 0
>
> while 1: # Infinite
> t = stopwatch.Timer()
> check = self.LastModified(self.arrayList)
> if check != modified or reset >= self.reset: # Rather than dumping our list and rereading
> if check != modified:
> print ('Array list modification detected. ',end="")
> if reset >= self.reset:
> print ('Reset triggered. We are {:1.2f} minutes past target threshold. '.format((reset - self.reset) / 60),end="")
> reset = 0
> print('Refreshing list...')
> expand = []
> expanded = []
> extend = []
> f = open(self.arrayList, 'r') # every loop we will shortcut this by checking
> for line in f: # the 'Last Modified' time stamp. If any changes
> line = line.strip() # take place we can assume the list was updated
> pingres = os.system('ping -c 1 -W 1 -t 255 ' + line + ' > /dev/null 2>&1')
> if pingres == 0 : # ifalive
> if line.startswith(self.housing):
> extend.append(line)
> elif (line[0].isdigit() or line[0].lower() <= 'i'):
> expand.append(line)
> else:
> expanded.append(line)
> modified = check
> f.close()
>
> sys.stdout.write('Running. . .\n')
> sys.stdout.flush()
>
> if self.verbose:
> sys.stdout.write('\x1b[2K\r')
>
> if len(expand) > 0 :
> self.Query(expand, 'expand', var)
>
> if len(expanded) > 0 :
> self.Query(expanded, 'expanded', var)
>
> if len(extend) > 0 :
> self.Query(extend, 'extend', var)
>
> t.stop()
> gc.collect()
> #if self.verbose == 1 :
> # print ('Sleeping.')
> print ('\rRan at', time.strftime('%c').strip(), 'for {:1.2f} seconds.'.format(t.elapsed))
> for remaining in range(self.sleep, 0, -1):
> sys.stdout.write(' \r')
> sys.stdout.write('{:2d} seconds remaining'.format(remaining))
> sys.stdout.flush()
> time.sleep(1)
> sys.stdout.write('\x1b[2K\r')
> sys.stdout.flush()
> reset += self.sleep + t.elapsed
>
> def LastModified(self,filename):
> try:
> t = os.path.getmtime(filename)
> except ValueError as e:
> logger.error(e)
> raise
> return datetime.datetime.fromtimestamp(t)
>
> def WriteOut(self,device,r):
> tofile = device + '.csv'
> f = open(tofile, 'a')
> f.write(datetime.datetime.fromtimestamp(time.time()).strftime('%a %d %b %Y %H:%M:%S %p %Z'))
> f.write(',' + r)
> f.write('\n')
> f.close()
>
> @memprof(plot = True)
> def SetUpIndex(self):
> index = 'xirrus-'+time.strftime("%Y.%m.%d")
> type = 'StationsPerIAP'
> check_exists=requests.get('http://localhost:9200/'+index)
> DEAD = 404 # error code 404: not found
> if check_exists.status_code == DEAD: # if the index doesn't exist MAKE IT...and set up the mapping as well!
> print("another day, another index.")
> type_mapping={
> "mappings" : {
> type: {
> "properties": {
> "@timestamp": {
> "type": "date",
> "format": "dateOptionalTime"
> },
> "building": {
> "type": "string",
> "index": "not_analyzed"
> },
> "logSource": {
> "type": "string",
> "index": "not_analyzed"
> },
> 'logSourceIP':{
> "type": "string",
> "index": "not_analyzed"
> },
> 'model':{
> "type":"string",
> "index": "not_analyzed"
> },
> "timestamp": {
> "type":"string",
> "index": "not_analyzed"
> },
> "radioSlots":{
> "type":"integer"
> }
> }
> }
> }
> }
> r=requests.put("http://localhost:9200/"+index,data=json.dumps(type_mapping))
> print("index creation:",r.json()['acknowledged'])
> del r
>
> return index+"/"+type
>
> @memprof(plot = True)
> def ToKibana(self,device,SNMP_response,index_api):
> #print(datetime.datetime.fromtimestamp(time.time()).strftime('%a %d %b %Y %H:%M:%S %p %Z'))
> #print(datetime.datetime.fromtimestamp(time.time()).strftime("%Y-%m-%dT%H:%M:%S.000Z"))
> radio_slots=(len(SNMP_response)-1)/2 # this would change if we collected more data
> single_doc = {}
> single_doc["logSource"]=device
> single_doc["logSourceIP"]=str(gethostbyname(device)) #https://{{value}}:1657
> single_doc["building"]=device.split("-")[0]
> single_doc["timestamp"]=datetime.datetime.fromtimestamp(time.time()).strftime('%a %d %b %Y %H:%M:%S %p')#add+6*3600 to offset the UTC by 6 hours
> single_doc["@timestamp"]=datetime.datetime.fromtimestamp(time.time()+6*3600).strftime("%Y-%m-%dT%H:%M:%S.000Z")
> single_doc["model"]=SNMP_response[0].split(',')[0]
> single_doc["radioSlots"]=radio_slots
>
> for x in range(1,radio_slots+1):
> band= "2.4GHz" if int(SNMP_response[x+radio_slots]) <= 11 else "5GHz"
> if int(SNMP_response[x+radio_slots]) != 0: #if wireless channel is not equal to zero
> single_doc["IAP{}".format(x)]=int(SNMP_response[x])
>
> r = requests.post("http://localhost:9200/"+index_api,data=json.dumps(single_doc))
> if self.verbose == 1:
> print ('snmp responce:',SNMP_response)
> del single_doc
> return r.json()['created']
> #print (device , json.dumps(single_doc))
>
> @memprof(plot = True)
> def Query(self, arrays, community_str, var):
> session = SnmpSession ( timeout = 1000000 ,
> retries = 1,
> verbose = self.verbose,
> oidlist = var,
> targets = arrays,
> community = community_str
> )
>
> #results = session.multiwalk(mode = 'bulkwalk')
> results = session.seqwalk()
> del session
>
> last_host = None
> hosts = [] # the list of hosts
> host_responses = [] # the list of responses for each host
> responses = []
> for output in results:
> if output.val is not None:
> if output.hostname != last_host: # new host
> if last_host: # only append host_responses after a new host
> host_responses.append(responses)
> hosts.append(output.hostname)
> responses = [output.val] # start the new list of responses
> last_host = output.hostname
> else: # same host, append the response
> responses.append(output.val)
> host_responses.append(responses)
> es_index=self.SetUpIndex()
> doc_uploaded = 0
> doc_failed = 0
> for host, responses in zip(hosts, host_responses):
> #self.WriteOut(host, ','.join(responses))
> didUpload = self.ToKibana(host, responses,index_api=es_index)
> if didUpload == True:
> doc_uploaded += 1
> else:
> doc_failed += 1
>
> print ('Document Upload Success Rate:',100*doc_uploaded/(doc_uploaded+doc_failed),'%')
> del doc_uploaded
> del doc_failed
> del didUpload
> del hosts
> del host_responses
> del responses
> del results
>
>
> def check_zero(value):
> ivalue = int(value)
> if ivalue < 1:
> raise argparse.ArgumentTypeError("%s is less than 1" % value)
> return ivalue
>
> if __name__ == '__main__':
> os.chdir(os.path.dirname(sys.argv[0]))
> parser = argparse.ArgumentParser(description='SNMPBULKWALK with definable hostname file and target OID')
> parser.add_argument('file', metavar='FILENAME', help='Name of file that contains target hosts')
> parser.add_argument('-o', '--OID', metavar='OID', default='iapNumStations', help='OID as translated to human-readable format')
> parser.add_argument('-s', '--sleep', metavar='seconds', type=check_zero, help='Time between querying list of devices', default=180)#change back to 120 or whatever
> parser.add_argument('-v', '--verbose', metavar='\b', help='Enable verbose messages', default=0, const=1, action='store_const')
> parser.add_argument('-r', '--reset', metavar='minutes', type=check_zero, help='Period of time before checking if \'dead\' hosts are now online.', default=180)
> args = parser.parse_args()

> f = Front().init(arrayList=args.file, verbose=args.verbose, sleep=args.sleep, reset=args.reset) #desiredOID=args.OID,

--