Flushing out data from Popen buffer

[abc79721]

I am working on a python script that reads data by tailing a file and then puts in a different file. The script works in a time bound manner and eventually flushes out the data from the buffer when the ENDTIME is reached. However there has been a mismatch in the source and target file in terms of size.

Following is a snippet:

self.read_size = 2048
self.tail_buffer = 2048

# start the file tail
cmd = '%s -f -o %s %s' % (self.jtailcmd, offset, self.source_journal)
self.logger.debug('[%s] starting FILETail' % self.getName())
try:
self.jtail = popen2.Popen3(cmd, bufsize = self.tail_buffer)
self.jtail.tochild.close()
out = self.jtail.fromchild
outfd = self.jtail.fromchild.fileno()
flags = fcntl.fcntl(outfd, fcntl.F_GETFL)
fcntl.fcntl(outfd, fcntl.F_SETFL, flags | os.O_NONBLOCK)
except:
message = '[%s] error reading file' % self.getName()
self.logger.error(message)
self.logger.error('[%s] %s: %s' % \
(self.getName(), sys.exc_info()[0], sys.exc_info()[1]))
send_alert('AE', message)
self.sleep(60)
self.close_tail()
self.close_ssh()
And then eventually it flushes out the data:

try:
[i, o, e] = select.select([outfd], [], [], 1)
if i:
data = out.read(self.read_size)
else:
data = None
except:
message = '[%s] error reading file' % self.getName()
self.logger.error(message)
self.logger.error('[%s] %s: %s' % \
(self.getName(), sys.exc_info()[0], sys.exc_info()[1]))
send_alert('AE', message)
self.close_tail()
self.close_ssh()
self.sleep(60)
break
if data:
if self.sshcat.poll() != -1:
self.logger.error('[%s] connection error' % self.getName())
self.close_tail()
self.close_ssh()
break
try:
self.sshcat.tochild.writelines(data)
self.sshcat.tochild.flush()
except:
message = '[%s] error writing remote file' % self.getName()
While troubleshooting, I narrowed out the problem to tail_buffer size! By reducing the tail_buffer size , the script worked fine. I cant use subprocess(Python Version is 2.4.3)

I do not want to rely on tail_buffer size. Ideally the script should be independent of it!

Is there a way to flush data from the POPEN buffer ?