We are running some automation scripts using Python. In this, we need
to open and Excel workbook and keep appending the results to the same
cell or overwrite certain cells.
We do this using Python and importing xlutils, xlrd and xlwt packages.
While we do this, if the excel sheet has already some data in a cell
and we try to overwrite or append to it, we get the error shown below
("List index out of bound"). We have observed that if the cells in
which we try to write are empty, this error doesn't show up and the
data is correctly written to the cells !!
A sample script of what we try to do is below:
def main():
import os,sys
from xlrd import open_workbook,xldate_as_tuple #Import package for
Reading Excel
from xlutils.copy import copy #Import Package for Writing Excel
from xlutils.save import save
from xlwt import Workbook
# Open the excel work book
rb = open_workbook('Automation.xls',formatting_info=True)
wb = copy(rb)
ws = wb.get_sheet(0)
ws.write(0,0,"value1")
print "Wrote in 1st cell"
ws.write(1,0,"value2")
print "Wrote in 2nd cell"
wb.save('Automation.xls')
return
if __name__ == '__main__':
main()
The error we get is as follows:
Traceback (most recent call last):
File "tmp.py", line 23, in <module>
main()
File "tmp.py", line 11, in main
wb = copy(rb)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/
python2.6/xlutils/copy.py", line 13, in copy
w
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/
python2.6/xlutils/filter.py", line 812, in process
reader(chain[0])
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/
python2.6/xlutils/filter.py", line 66, in __call__
filter.cell(row_x,col_x,row_x,col_x)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/
python2.6/xlutils/filter.py", line 471, in cell
style = self.style_list[cell.xf_index]
IndexError: list index out of range
If you can help us get a solution to this, we would really appreciate
it.
Thanks in advance !
Exilant team.
According to your traceback, the error is happening on the above line.
This is well before you try to write or append, and there are no loops
in your code. In other words, your story is inconsistent.
Note that you read 'Automation.xls' and save your output as the same
name. Consider the possibility that you may have used a corrupted input
file. Please start with a fresh known tested input file. Please bracket
the above statement with print statements to assist in demonstrating
exactly what is happening. Please ensure that the code that you paste
into your next message is the exact code that you ran. Please make
available a copy of the input file.
> ws = wb.get_sheet(0)
>
> ws.write(0,0,"value1")
> print "Wrote in 1st cell"
> ws.write(1,0,"value2")
> print "Wrote in 2nd cell"
>
> wb.save('Automation.xls')
> return
>
> if __name__ == '__main__':
> main()
>
> The error we get is as follows:
>
> Traceback (most recent call last):
> File "tmp.py", line 23, in <module>
> main()
> File "tmp.py", line 11, in main
> wb = copy(rb)
Error happens in xlutils.copy ...
We are actually running the .py from a GUI testing tool called Squish
(www.froglogic.com). So when I run the same test.py file from the
Squish tool, it is corrupting the file when it tries to overwrite a
cell, only when there is formatting information on that sheet/any
sheet.
So I am not able to figure out how it is leading to an Index Error:
list index out of range.
Here is the script that I ran:
----------------------------------
def main():
import os,sys
sys.path.append('/work/Merlin/requiredPackages')
from xlrd import open_workbook,xldate_as_tuple #Import package for
Reading Excel pppt "\n Here\n"
from xlutils.copy import copy #Import Package for Writing Excel
from xlutils.save import save
from xlwt import Workbook
print "running test.py"
# Open the excel work book
rb = open_workbook('Sample.xls',formatting_info=True)
wb = copy(rb)
ws = wb.get_sheet(0)
ws.write(0,0,"value5")
print "Wrote in 1st cell"
ws.write(1,0,"value6")
print "Wrote in 2nd cell"
wb.save('Sample.xls')
return
if __name__ == '__main__':
main()
Here is the error I see:
------------------------------
ERROR Fri Jan 15 15:50:34 2010 /work/Merlin/requiredPackages/
xlutils/filter.py:473: Script Error
IndexError: list index out of range
Called from:
/work/Merlin/Jan11_Sow_FrameWork/tst_simpleExcel/test.py: 12
/work/Merlin/requiredPackages/xlutils/copy.py: 13
/work/Merlin/requiredPackages/xlutils/filter.py: 814
/work/Merlin/requiredPackages/xlrd/xldate.py: 66
Please let me know how I can attache the excel sheet Sample.xls here.
It has nothing but some formatted cells with background colour etc in
cells other than 0,0 and 1,0. In cells 0,0 and 1,0 i have some sample
text which I try to overwrite everytime I run the script.
I am using Python 2.6, xlutils 1.4.1 and froglogic Squish 4.0.0. I
contacted the Squish people and they say there is no problem on Squish
side !?
Thanks and regards,
Exilant Team.
That's NOT what I meant by starting with a FRESH input xls file!
> and then run
> the test.py file in the terminal, there is no problem.
> If I run test.py file from the terminal without saving the .xls file,
I'm presuming that "in the terminal" and "from the terminal" mean the
same thing.
> I see the error. I guess this is expected.
>
> We are actually running the .py from a GUI testing tool called Squish
> (www.froglogic.com). So when I run the same test.py file from the
> Squish tool, it is corrupting the file when it tries to overwrite a
> cell, only when there is formatting information on that sheet/any
> sheet.
So are you saying that if you run it in the terminal it's OK but if you
run it with Squish it blows up?
> So I am not able to figure out how it is leading to an Index Error:
> list index out of range.
>
> Here is the script that I ran:
> ----------------------------------
>
> def main():
> import os,sys
> sys.path.append('/work/Merlin/requiredPackages')
This is new; why do you need this?
> from xlrd import open_workbook,xldate_as_tuple #Import package for
> Reading Excel pppt "\n Here\n"
> from xlutils.copy import copy #Import Package for Writing Excel
> from xlutils.save import save
> from xlwt import Workbook
> print "running test.py"
>
> # Open the excel work book
> rb = open_workbook('Sample.xls',formatting_info=True)
> wb = copy(rb)
I don't see the two print statements that I asked for :-(
> ws = wb.get_sheet(0)
> ws.write(0,0,"value5")
> print "Wrote in 1st cell"
> ws.write(1,0,"value6")
> print "Wrote in 2nd cell"
> wb.save('Sample.xls')
You're still overwriting the input file.
> return
>
> if __name__ == '__main__':
> main()
>
> Here is the error I see:
Where is the print statement output?
> ------------------------------
>
> ERROR Fri Jan 15 15:50:34 2010 /work/Merlin/requiredPackages/
> xlutils/filter.py:473: Script Error
> IndexError: list index out of range
> Called from:
> /work/Merlin/Jan11_Sow_FrameWork/tst_simpleExcel/test.py: 12
> /work/Merlin/requiredPackages/xlutils/copy.py: 13
> /work/Merlin/requiredPackages/xlutils/filter.py: 814
> /work/Merlin/requiredPackages/xlrd/xldate.py: 66
This is in effect the same traceback as before, with TWO exceptions
(1) the above line was .../xlutils/filter.py: 66
now it's .../xlrd/xldate.py: 66
Looks like Squish has had a brain seizure; xlutils.filter doesn't call
anything in xlrd.xldate. Your script does "from xlrd import ...,
xldate_as_tuple" but doesn't use it.
(2) previously the error line was reported as .../xlutils/filter.py,
line 471 but now it's line 473 (which contains "style = default_style"
(IndexError not possible)) ... different version?? you've been editing
.../xlutils/filter.py ?? Squish has had 2 brain seizures?
This means that it's still blowing up in "wb = copy(rb)", which is
absolutely nothing to do with the intent of the following
not-yet-excecuted code. So we've got precisely nowhere. Please do the
following:
1. abandon Squish; run the debug script from/in/at the
terminal/console/shell
2. abandon your Sample.xls; create a fresh one by typing into the Excel
UI. Upload it to the files section of this group (go to
http://groups.google.com/group/python-excel and click on the Files link)
3. put print statements before and after the "wb = copy(rb)" line
4. change the script so that it saves the output file with a new name
5. run the script from the shell
6. copy and paste (1) the contents of the script (2) the output (3)
traceback and error message into your next message.
> Please let me know how I can attache the excel sheet Sample.xls here.
see above
> It has nothing but some formatted cells with background colour etc in
> cells other than 0,0 and 1,0. In cells 0,0 and 1,0 i have some sample
> text which I try to overwrite everytime I run the script.
>
> I am using Python 2.6, xlutils 1.4.1 and froglogic Squish 4.0.0.
Python 2.6.WHAT? What versions of xlrd and xlwt?
> I
> contacted the Squish people and they say there is no problem on Squish
> side !?
Well, they would say that, wouldn't they?
Thanks a lot for all the suggestions you are giving us. Appreciate
it !
Yes, I started with a fresh input xls file which I now call
FreshSample.xls.
Yes, I was saying that if I run it in the terminal it's OK but if I
run it with Squish it blows up.
I am using sys.path.append('/work/Merlin/requiredPackages') because
the Squish tool was not picking up the packages (xlwt, xlrt and
xlutils) from /Library/Frameworks/Python.framework/Versions/2.6/lib/
python2.6. I have a copy of the packages under the path I am
appending to sys.path.
In the directory xlutils I see a version.txt which says 1.4.1. I did
not see a version.txt under xlwt or xlrd. So when I open init.py for
xlwt, I see the line - __VERSION__ = '0.7.2'
Similarly in directory xlrd, in the file init.py, I see __VERSION__ =
"0.7.1" # 2009-05-31
Python version I am using is 2.6.1 (r261:67515, Jul 7 2009,
23:51:51)
The system I am using is a MacBook with a Snow Leopard OS (10.6)
Sorry about the print statements. Will add it and run the next
time. :-)
The error I had pasted before is seen in the Squish tool.
>> (2) previously the error line was reported as .../xlutils/filter.py, line 471 but now it's line 473 (which contains "style = default_style"
>> (IndexError not possible)) ... different version?? you've been editing .../xlutils/filter.py ?? Squish has had 2 brain seizures?
Yes, we modified the filter.py to have a work around solution. This we
did to try and make it work when we ran the script from the Squish
tool. The modification we did was,
Original it was: Line 471 - if cell.xf_index is not None:
Modified lines:
471: # Work around to make it work in Squish
472: # if cell.xf_index is not None:
473: if cell.xf_index is not None and cell.xf_index < 100:
This is why the error showed up in 473 the next time. Since
our .xls file has about 30 rows right now, this work around fix when
uncommented helps us to avoid the error message.
As I was trying to say in my previous message the error does not occur
if I run the script in the terminal. But only occurs when I run the
same script from the Squish tool. So if I abandon the Squish tool, I
don't see the error and saving to the Excel file works fine.
Now, as per your suggestion I gave a new name to the file I was saving
to and did not overwrite the input file. This helped in solving the
problem when I run my test.py in the Squish tool !!!
But in our actual scenario, for each of the numerous test cases we
have, Squish opens the Automation.xls, takes the input data to the
particular test case from it, runs the test and saves the result to
the same Automation.xls. We cannot save it to a different file each
time as we will lose the results of the previous test cases each time
we save it to a new one. Please note that each time we are using
open_workbook API. Since we did not find any API to close the
workbook, we just save and proceed in a loop to run the next test
case.
I have uploaded the FreshSample.xls and SavedSample.xls to the place
you have mentioned and here is my test.py that I run both in the
terminal and the Squish tool.
test.py:
def main():
import os,sys
sys.path.append('/work/Merlin/requiredPackages')
from xlrd import open_workbook,xldate_as_tuple
from xlutils.copy import copy #Import Package for Writing Excel
from xlutils.save import save
from xlwt import Workbook
print "running test.py"
# Open the excel work book
rb = open_workbook('FreshSample.xls',formatting_info=True)
print 'Before calling wb = copy rb ...'
wb = copy(rb)
print 'After calling wb = copy rb ...'
ws = wb.get_sheet(0)
ws.write(0,0,"Modified value for Cell 1")
print "Wrote in 1st cell"
ws.write(1,0,"Modified value for Cell 2")
print "Wrote in 2nd cell"
wb.save('SavedSample.xls')
return
if __name__ == '__main__':
main()
-----------------------------------------------------------------------------------------
Output:
running test.py
Before calling wb = copy rb ...
After calling wb = copy rb ...
Wrote in 1st cell
Wrote in 2nd cell
running test.py
Before calling wb = copy rb ...
After calling wb = copy rb ...
Wrote in 1st cell
Wrote in 2nd cell
(not sure why I am seeing 2 sets of print statements ?!)
-----------------------------------------------------------------------------------------
Error:
I don't see any errors when I save it to a new file if I run in
Squish tool or in terminal.
I see the following error in Squish log (ONLY IF I RUN IN
SQUISH and not in the terminal) only when I try to overwrite the input
file which has some formatted data(which is a requirement for the
actual test scenario). I just now also figured out that that error
occurs when there are merged cells anywhere in the workbook. (I
removed the merged cells and there is no error !!)
ERROR Sat Jan 16 20:48:57 2010 /work/Merlin/requiredPackages/
xlutils/filter.py:473: Script Error
IndexError: list index out of range
Called from:
/work/Merlin/Jan11_Sow_FrameWork/tst_simpleExcel/test.py: 13
/work/Merlin/requiredPackages/xlutils/copy.py: 13
/work/Merlin/requiredPackages/xlutils/filter.py: 814
/work/Merlin/requiredPackages/xlrd/xldate.py: 66
-----------------------------------------------------------------------------------------
Thanks and regards,
Exilant Team.
> UI. Upload it to the files section of this group (go tohttp://groups.google.com/group/python-exceland click on the Files link)
Thanks,
Exilant.
> > UI. Upload it to the files section of this group (go tohttp://groups.google.com/group/python-excelandclick on the Files link)
>
> > 3. put print statements before and after the "wb = copy(rb)" line
>
> > 4. change the script so that it saves the output file with a new name
>
> > 5. run the script from the shell
>
> > 6. copy and paste (1) the contents of the script (2) the output (3)
> > traceback and error message into your next message.
>
> > > Please let me know how I can attache the excel sheet Sample.xls here.
>
> > see above
>
> > > It has nothing but some formatted cells with background colour etc in
> > > cells other than 0,0 and 1,0. In cells 0,0 and 1,0 i have some sample
> > > text which I try to overwrite everytime I run the script.
>
> > > I am using Python 2.6, xlutils 1.4.1 and froglogic Squish 4.0.0.
>
> > Python 2.6.WHAT? What versions of xlrd and xlwt?
>
> > > I
> > > contacted the Squish people and they say there is no problem on Squish
> > > side !?
>
> ...
>
> read more »
So why are you asking here about the Squish problem instead of utilising
the Squish support to which your LICENCE FEE entitles you?
> I am using sys.path.append('/work/Merlin/requiredPackages') because
> the Squish tool was not picking up the packages (xlwt, xlrt and
> xlutils) from /Library/Frameworks/Python.framework/Versions/2.6/lib/
> python2.6. I have a copy of the packages under the path I am
> appending to sys.path.
>
> In the directory xlutils I see a version.txt which says 1.4.1. I did
> not see a version.txt under xlwt or xlrd. So when I open init.py for
> xlwt, I see the line - __VERSION__ = '0.7.2'
> Similarly in directory xlrd, in the file init.py, I see __VERSION__ =
> "0.7.1" # 2009-05-31
>
> Python version I am using is 2.6.1 (r261:67515, Jul 7 2009,
> 23:51:51)
> The system I am using is a MacBook with a Snow Leopard OS (10.6)
>
> Sorry about the print statements. Will add it and run the next
> time. :-)
What is so humourous about that promise? The fact that you didn't
promise to show the output of the print statements when the error
occurred (see below)?
> The error I had pasted before is seen in the Squish tool.
>
>>> (2) previously the error line was reported as .../xlutils/filter.py, line 471 but now it's line 473 (which contains "style = default_style"
>>> (IndexError not possible)) ... different version?? you've been editing .../xlutils/filter.py ?? Squish has had 2 brain seizures?
>
> Yes, we modified the filter.py to have a work around solution.
But didn't mention it :-(
Have you fiddled with any other code in xlrd, xlwt or xlutils?
> This we
> did to try and make it work when we ran the script from the Squish
> tool. The modification we did was,
>
> Original it was: Line 471 - if cell.xf_index is not None:
>
> Modified lines:
> 471: # Work around to make it work in Squish
> 472: # if cell.xf_index is not None:
> 473: if cell.xf_index is not None and cell.xf_index < 100:
>
> This is why the error showed up in 473 the next time. Since
> our .xls file has about 30 rows right now, this work around fix when
> uncommented helps us to avoid the error message.
>
>
> As I was trying to say in my previous message the error does not occur
> if I run the script in the terminal. But only occurs when I run the
> same script from the Squish tool. So if I abandon the Squish tool, I
> don't see the error and saving to the Excel file works fine.
>
> Now, as per your suggestion I gave a new name to the file I was saving
> to and did not overwrite the input file. This helped in solving the
> problem when I run my test.py in the Squish tool !!!
>
> But in our actual scenario,
Why didn't you tell this story in your *FIRST* message?
> for each of the numerous test cases we
> have, Squish opens the Automation.xls,
Are you sure that Squish re-opens the file for each of the second,
third, ... tests?
> takes the input data to the
> particular test case from it,
Does Squish close the file at this stage?
> runs the test
I presume you mean "runs the test.py file".
This will:
(1) use xlrd.open_workbook() to
(1a) file_handle = open("Automation.xls", "rb")
(1b) extract the Workbook stream from the file
(1c) file_handle.close()
(1d) build and return an xlrd.Book object
(2) use xlutils to copy the xlrd.Book to an xlwt.Workbook object
(3) modify the xlwt.Workbook object
(4) call Workbook_object.save("Automation.xls")
(4a) use file_handle = open("Automation.xls", "wb")
(4b) a few file_handle.write(<<packed contents of Workbook_object>>)
(4c) file_handle.close()
and saves the result to
> the same Automation.xls.
Is this bracketed by open/close? Same name or same file? Note that step
(4a) above will blow away any existing file with that name.
> We cannot save it to a different file each
> time as we will lose the results of the previous test cases each time
> we save it to a new one.
For debug purposes, change your script so that it does BOTH of
(1) save as "Automation.xls" (as you say you require for proper operation)
AND
(2) save as "Automation <<microsecond-precision timestamp>>.xls"
so that we get a record of what is actually being saved.
Run the script under conditions that cause the error to happen.
Provide each of the following UNMOLESTED files:
(a) backup copy of Automation.xls taken immediately before running the
script
(b) copy of Automation.xls taken immediately after the error happens
(c) ALL "Automation <<microsecond-precision timestamp>>.xls" files that
were created
> Please note that each time we are using
> open_workbook API. Since we did not find any API to close the
> workbook, we just save and proceed in a loop to run the next test
> case.
The "open" in "open_workbook" is a bit of a misnomer. As you see from
step (1) above, the file handle is not left open when open_workbook returns.
The whole idea of having test scheduling info and test data and test
results in the one XLS file which both the test runner and the test
script are updating (or, worse, DELETING and REPLACING) strikes me as
being very iffy. Are you sure that this is a/the usual and supported way
of using Squish?
> I have uploaded the FreshSample.xls and SavedSample.xls to the place
> you have mentioned and here is my test.py that I run both in the
> terminal and the Squish tool.
Here are the first few lines of the output of the ov command of the
runxlrd script when applied to your SavedSample.xls:
"""
=== File: SavedSample.xls ===
Open took 0.00 seconds
BIFF version: 8; datemode: 0
codepage: 1200 (encoding: utf_16_le); countries: (1, 1)
Last saved by: u'Srilatha Sriram'
"""
The last two items indicate that the file has been MOLESTED by a user
(presumably the one named) after it was created by xlwt.Workbook.save()
thus it has no forensic value.
I'm not sure why either :-)
Possibilities (1) squish is doing something like "tee"ing stdout
(2) Squish is running the test twice
Have you investigated this? asked Squish support? Tried printing the
time (to (say) millisecond resolution) in your first print statement?
It is important to be sure (a) whether the error occurs on the first or
second test (b) exactly where the error occurs. Please put the time on
the first print statement and SHOW THE PRINT OUTPUT WHEN AN ERROR OCCURS
>
> -----------------------------------------------------------------------------------------
>
> Error:
> I don't see any errors when I save it to a new file if I run in
> Squish tool or in terminal.
>
> I see the following error in Squish log (ONLY IF I RUN IN
> SQUISH and not in the terminal) only when I try to overwrite the input
> file which has some formatted data(which is a requirement for the
> actual test scenario).
As I have tried to point out twice before, what you try to do is
irrelevant because the error is (believed to be)? happening BEFORE the
code which does any overwriting.
If you mean that the error occurs when the input file has some
"formatted data" but doesn't occur when the input file has no "formatted
data", then please say so!
> I just now also figured out that that error
> occurs when there are merged cells anywhere in the workbook. (I
> removed the merged cells and there is no error !!)
>
> ERROR Sat Jan 16 20:48:57 2010 /work/Merlin/requiredPackages/
> xlutils/filter.py:473: Script Error
> IndexError: list index out of range
> Called from:
> /work/Merlin/Jan11_Sow_FrameWork/tst_simpleExcel/test.py: 13
> /work/Merlin/requiredPackages/xlutils/copy.py: 13
> /work/Merlin/requiredPackages/xlutils/filter.py: 814
> /work/Merlin/requiredPackages/xlrd/xldate.py: 66
I say again: THE ABOVE LINE IS SYMPTOMATIC OF A BUG -- See below
Where's the output from the print statements???????
Please read this message carefully, answer ALL the questions, change
your script, run it again, show all the output, post a zip of your
script and all the requested files.
>
> -----------------------------------------------------------------------------------------
>
> Thanks and regards,
> Exilant Team.
>
>
> On Jan 15, 5:37 pm, John Machin <sjmac...@lexicon.net> wrote:
>> On 15/01/2010 9:34 PM, Exilant wrote:
>>
>>> Here is the error I see:
>> Where is the print statement output?
YOU DIDN'T ADDRESS THIS ISSUE. THE WHOLE POINT OF THE PRINT STATEMENTS
IS TO SHOW HOW FAR YOUR SCRIPT HAD PROGRESSED BEFORE THE ERROR HAPPENED
-- TO GIVE SOME BACKUP TO THE *DODGY* TRACEBACK MECHANISM.
>>
>>> ------------------------------
>>> ERROR Fri Jan 15 15:50:34 2010 /work/Merlin/requiredPackages/
>>> xlutils/filter.py:473: Script Error
>>> IndexError: list index out of range
>>> Called from:
>>> /work/Merlin/Jan11_Sow_FrameWork/tst_simpleExcel/test.py: 12
>>> /work/Merlin/requiredPackages/xlutils/copy.py: 13
>>> /work/Merlin/requiredPackages/xlutils/filter.py: 814
>>> /work/Merlin/requiredPackages/xlrd/xldate.py: 66
>> This is in effect the same traceback as before, with TWO exceptions
>>
>> (1) the above line was .../xlutils/filter.py: 66
>> now it's .../xlrd/xldate.py: 66
>>
>> Looks like Squish has had a brain seizure; xlutils.filter doesn't call
>> anything in xlrd.xldate. Your script does "from xlrd import ...,
>> xldate_as_tuple" but doesn't use it.
YOU DIDN'T ADDRESS THE ABOVE ISSUE.
Also, did it not occur to you to remove the unused xldate_as_tuple from
the import statement in order to see the effect on the traceback?
Sorry, I didn't see this follow-up until after I'd sent my reply to the
previous message.
My hypothesis: Squish is running the test twice. First time is OK.
Second time, something goes wrong because one or more of (1) contention
over same file (2) xlutils stuffs up merged cells (3) Squish stuffs up
merged cells (4) something else
Please do everything that I "suggested" in the previous message, PLUS
add some debugging prints into filter.py just before the error-causing line:
if cell.xf_index is not None:
print "==== DEBUG: xlutils.filter (Basewriter.cell) ===="
print self.rdsheet.name
print rdrowx,rdcolx,wtrowx,wtcolx
print repr(cell)
print len(self.style_list)
print self.merged_cell_top_left_map
print self.merged_cell_already_set
print "==== end of debug output ===="
style = self.style_list[cell.xf_index]