coverage xml attempts to tokenize non-Python files

[Nathan Yergler]

I'm working on configuring our build environment to use coverage with
Hudson, and have run into an issue. We have some code that uses
Jinja2 to load and render a template, and a test that, well, tests
that. I can run nosetests --with-coverage just fine, but when I
attempt to generate the XML report, I get the following exception:

File "/usr/lib/python2.6/tokenize.py", line 346, in generate_tokens
("<tokenize>", lnum, pos, line))
File "<tokenize>", line 4
<img src="http://i.creativecommons.org/l/zero/1.0/88x31.png"
style="border-style: none;" alt="CC0" />
^
IndentationError: unindent does not match any outer indentation level

(see full traceback at the end). Any thoughts? Happy to provide
additional details if needed.

Thanks,

Nathan

nyergler@marten:~/p/cc.license$ ./bin/coverage xml
Traceback (most recent call last):
File "./bin/coverage", line 30, in <module>
coverage.main()
File "/home/nyergler/p/cc.license/eggs/coverage-3.3.1-py2.6-linux-
i686.egg/coverage/cmdline.py", line 601, in main
status = CoverageScript().command_line(argv)
File "/home/nyergler/p/cc.license/eggs/coverage-3.3.1-py2.6-linux-
i686.egg/coverage/cmdline.py", line 510, in command_line
self.coverage.xml_report(outfile=outfile, **report_args)
File "/home/nyergler/p/cc.license/eggs/coverage-3.3.1-py2.6-linux-
i686.egg/coverage/control.py", line 390, in xml_report
morfs, omit_prefixes=self.config.omit_prefixes, outfile=outfile
File "/home/nyergler/p/cc.license/eggs/coverage-3.3.1-py2.6-linux-
i686.egg/coverage/xmlreport.py", line 55, in report
self.report_files(self.xml_file, morfs,
omit_prefixes=omit_prefixes)
File "/home/nyergler/p/cc.license/eggs/coverage-3.3.1-py2.6-linux-
i686.egg/coverage/report.py", line 57, in report_files
report_fn(cu, self.coverage._analyze(cu))
File "/home/nyergler/p/cc.license/eggs/coverage-3.3.1-py2.6-linux-
i686.egg/coverage/control.py", line 309, in _analyze
return Analysis(self, it)
File "/home/nyergler/p/cc.license/eggs/coverage-3.3.1-py2.6-linux-
i686.egg/coverage/results.py", line 30, in __init__
self.statements, self.excluded = self.parser.parse_source()
File "/home/nyergler/p/cc.license/eggs/coverage-3.3.1-py2.6-linux-
i686.egg/coverage/parser.py", line 184, in parse_source
self._raw_parse()
File "/home/nyergler/p/cc.license/eggs/coverage-3.3.1-py2.6-linux-
i686.egg/coverage/parser.py", line 90, in _raw_parse
for toktype, ttext, (slineno, _), (elineno, _), ltext in tokgen:
File "/usr/lib/python2.6/tokenize.py", line 346, in generate_tokens
("<tokenize>", lnum, pos, line))
File "<tokenize>", line 4
<img src="http://i.creativecommons.org/l/zero/1.0/88x31.png"
style="border-style: none;" alt="CC0" />
^
IndentationError: unindent does not match any outer indentation level

[Ned Batchelder]

Hi Nathan,

This error is reported at
http://bitbucket.org/ned/coveragepy/issue/82/tokenerror-when-generating-html-report
. But I would love to have a small test case that reproduces it. I'm
not familiar with Jinja, and don't yet understand how it reports an HTML
file as the file being executed.

--Ned.

[Nathan R. Yergler]

Ah, sorry I didn't see that previously. I've added a small testcase +
instructions to that bug which reproduces the issue.

I don't pretend to be a Jinja expert (or even all that knowledgeable),
but I wonder if it has to do with Jinja "compiling" templates to
Python byte code?

Thanks,

Nathan

[Ned Batchelder]

I've never used Jinja before, so thanks for the test case. I debugged
through it: Jinja does compile the html into Python code, and uses the
name of the HTML file as the filename in the compiled code. I thought
there might be a slim chance that coverage could get line numbers in the
original HTML file, and do something cool like highlight lines in the
html file as covered or not. But it's not so: the line numbers are from
the compiled .py code, but with the html filename, so they're useless.

I'm going to fix this by having the trace function ignore .html files.

--Ned.