svn timeline problems

[Sergio A. Kessler]

hi Wez,

it seems the timeline is not working very well with svn...

I implemented the post-commit hook


$ cat /home/sak/svn/hooks/post-commit
#!/bin/sh
/usr/bin/php /opt/mtrack/bin/svn-commit-hook post $1 $2 /opt/mtrack/config.ini

I made a very simple changue in my wc and commited

the timeline is showing me:

(In 730) update header.php update header.php
14:04 default/sak by sak

my message was "update header.php"
it is displaying repeated ?

BUT the real problem is the changeset, mtrack timeline is showing:

			Index: header.php
			===================================================================
			--- header.php (revision 729)
			+++ header.php (revision 730)
		#	@@ -20,7 +20,7 @@
20	20	#	</td>
21	21	#	</tr>
22	22	#	<tr class="noprint">
23		#
	23	#
24	24	#	</tr>
25	25	#	</table>
26	26	#	</center>

but the changue was

+++ Z:/work/perio/fotocop/header.php dom abr 29 15:03:50 2012

@@ -20,7 +20,7 @@

       </td>

     </tr>

     <tr class="noprint">

-      <td><h1>TU Centro de Estudiantes La Walsh Conducción</h1></td>

+      <td><h1>TU Centro de Estudiantes, La Walsh Conducción</h1></td>

     </tr>

 </table>

 </center> 

[Sergio A. Kessler]

just in case:

$ svn --version

svn, version 1.6.12 (r955767)

   compiled Jun 22 2010, 11:47:19

[Sergio A. Kessler]

more data...

it seems the subversion in the server is older than my client tortoiseSvn (compiled with 1.7.3)

[sak@celeron fotocop]$ svn info

svn: The path '.' appears to be part of a Subversion 1.7 or greater

working copy rooted at '/home/sak/work/perio'.

Please upgrade your Subversion client to use this working copy.

so, I upgraded subversion to 1.7.4 on the server, now:

[sak@celeron fotocop]$ svn info

Path: .

Working Copy Root Path: /home/sak/work/perio

URL: svn+ssh://s...@192.168.1.2/home/sak/svn/perio/fotocop

Repository Root: svn+ssh://s...@192.168.1.2/home/sak/svn

Repository UUID: f21529d9-f819-0410-98fc-c74a1880cf43

Revision: 695

Node Kind: directory

Schedule: normal

Last Changed Author: sak

Last Changed Rev: 693

Last Changed Date: 2011-10-19 12:19:59 -0300 (Wed, 19 Oct 2011)

then, tried to make and commit another change, timeline shows:

(In 731) here we try with un updated subversion in the server here we try with un updated subversion in the server

clearly, the message is repeated...

clicking in the changeset, again, show a empty change:

			Index: header.php
			===================================================================
			--- header.php (revision 730)
			+++ header.php (revision 731)

and the change was:

===================================================================

--- Z:/work/perio/fotocop/header.php (revision 730)

+++ Z:/work/perio/fotocop/header.php (revision 731)

@@ -20,7 +20,7 @@

       </td>

     </tr>

     <tr class="noprint">

-      <td><h1>TU Centro de Estudiantes, La Walsh Conducción</h1></td>

+      <td><h1>TU Centro de Estudiantes La Walsh Conducción</h1></td>

     </tr>

 </table>

 </center>

[Wez Furlong]

I think the doubling issue is due to code in the timeline handling that tries to make sure that we didn't miss the revision log for changes that don't use the "refs #123" ticket command.

The diff thing is something I've not seen before, but wonder if it is somehow related to the accented characters in that commit.  What is the locale set to on that system?

--Wez.

--
The mtrack software development tracker
To post to this group, send email to mtr...@googlegroups.com
To unsubscribe, send email to mtrack+un...@googlegroups.com
For more options: http://groups.google.com/group/mtrack?hl=en
#mtrack on the freenode IRC network

[Sergio A. Kessler]

# locale

LANG=en_US.ISO8859-1

LC_CTYPE="en_US.ISO8859-1"

LC_NUMERIC="en_US.ISO8859-1"

[...]

can I tell mtrack that the repo is in ISO8859-1 ?

[Wez Furlong]

mtrack wants everything to be UTF-8; can you try this?

service httpd stop

rm -f $VARDIR/cmdcache/*

export LANG=en_US.UTF-8

service httpd start

where $VARDIR corresponds to the vardir you have configured for mtrack (it is often "var" in your mtrack checkout, unless you've explicitly moved it somewhere else).

Then try viewing that diff again?

--Wez.

[Sergio A. Kessler]

ok, I did:

# service httpd stop

# rm -f /opt/mtrack/cmdcache/*

# export LANG=en_US.UTF-8

# service httpd start

but nop, nothing change, mtrack did not show the lines changed...

and yes, is because the accented character,

I tried another commit with only ascii chars, and mtrack did show the lines...

but I cannot change the repo...

[Wez Furlong]

I've done some digging, and this is a problem that may have deeper ramifications, but that does have a simple resolution for you:

% hg diff inc/web.php

diff --git a/inc/web.php b/inc/web.php

--- a/inc/web.php

+++ b/inc/web.php

@@ -1110,7 +1110,7 @@

     $anchor = $abase . '.' . $nlines;

     $row .= "<td class='linelink'><a name='$anchor'></a><a href='#$anchor' titl

 

-    $line = htmlspecialchars($line, ENT_QUOTES, 'utf-8');

+    $line = htmlspecialchars($line);

     $row .= "<td class='line' width='100%'>$line</td></tr>\n";

     $html .= $row;

 

What's happening is that the mtrack diff renderer assumes that the content is in utf-8 and invokes htmlspecialchars with that charset.  In this case, the data is not valid UTF-8 and so htmlspecialchars returns an empty string.  The above change pretends that the diff is in an 8-bit encoding and passes it through.  I believe that this is generally safe for most purposes, but does raise the issue with how mtrack treats data that is not actually in UTF-8.

I also have a patch for the timeline issue that you mentioned; I think I'll go ahead and commit and apply both of these in a few moments.

--Wez.

[Sergio A. Kessler]

hmm, good news, bad news...

I've updated mtrack with the 2 recent changesets:

# hg update

3 files updated, 0 files merged, 0 files removed, 0 files unresolved

# rm -f /opt/mtrack/cmdcache/*

# service httpd restart  (just in case)

1) the repeated text in the timeline is fixed, thanks !

2) but the 

mtrack/changeset.php/default/sak/731

is still showing empty lines...

what I see, is that there are many more 

htmlentities($foobar, ENT_QUOTES, 'utf-8')

in inc/web.php (don't know the relevance in this case)

[Wez Furlong]

On Apr 29, 2012, at 8:57 PM, Sergio A. Kessler wrote:

1) the repeated text in the timeline is fixed, thanks !

Great!

2) but the 

mtrack/changeset.php/default/sak/731

is still showing empty lines...

:-/

Is your diff bigger than the example you included earlier?

I don't see the problem in my installation with the fix in place.

I wonder what else is different about our environments?

I'm running the OS X PHP 5.3.8; what PHP version do you have?

If you're looking for a workaround, you could try commenting out that line completely.

If you still see blanks after that, presumably something else is at fault.

what I see, is that there are many more 

htmlentities($foobar, ENT_QUOTES, 'utf-8')

in inc/web.php (don't know the relevance in this case)

I don't believe that any of those are applicable here, at least, not based on the details you've provided so far.

--Wez.

[Sergio A. Kessler]

>
> 2) but the
> mtrack/changeset.php/default/sak/731
> is still showing empty lines...
>
>
> :-/
>
> Is your diff bigger than the example you included earlier?
> I don't see the problem in my installation with the fix in place.
> I wonder what else is different about our environments?

nop, it's the same diff, I mean, I did not made another change and
commit, I'm trying to see the *same* diff trough mtrack...
I did
rm -f /opt/mtrack/cmdcache/*
just in case is cached, but nope...

I will try to make a different change...

> I'm running the OS X PHP 5.3.8; what PHP version do you have?

linux
# rpm -q php53
php53-5.3.3-5.el5

> If you're looking for a workaround, you could try commenting out that line completely.
> If you still see blanks after that, presumably something else is at fault.

ok, I modified the code, thus becoming:

// deliberately don't inform it of the charset; we have no idea and we
// only really care about the obvious HTML metacharacters, not the entities
//$line = htmlspecialchars($line);

$row .= "<td class='line' width='100%'>$line</td></tr>\n";

and exactly the same happens,
changeset.php/default/sak/731
is displaying an empty diff (a empty red row, and a green one)

[Sergio A. Kessler]

ok, with a new commit, the changed lines appears,
with a bigger font and the accented char as a question block :-/

image attached...

[Sergio A. Kessler]

ok, I uncommented the htmlspecialchars line, and the bigger font is
back to normal,
but the blocky character is still there...

I wonder how trac resolves this, because is outputting utf8,
but show the iso8859-1 files and diffs just fine...

btw, I was convinced that mtrack was just asking svn for the diff on-the-fly,
but it seems mtrack is storing the diff in the database...


On Sun, Apr 29, 2012 at 23:55, Sergio A. Kessler

[Sergio A. Kessler]

ok, I fixed the issue changing htmlspecialchars() by htmlentities()

so, the line:
$line = htmlspecialchars($line);
becomes:
$line = htmlentities($line);

and now, the diff is displaying just rigth... ;-)

(thanks to w3c that almost all iso88591 have html entities)

but this is not the approach of Trac, because the source page of trac
doesn't have the accented characters as html entities, they show it as
utf8...



On Mon, Apr 30, 2012 at 00:14, Sergio A. Kessler

[Wez Furlong]

On Apr 29, 2012, at 11:14 PM, Sergio A. Kessler wrote:

> btw, I was convinced that mtrack was just asking svn for the diff on-the-fly,
> but it seems mtrack is storing the diff in the database...

Yes; mtrack caches the output for this fairly strongly, because a changeset is unlikely to change. The cmdcache dir holds the cached data; are you sure that you were rm'ing it before?

Generally speaking, the vardir is the "var" dir of your mtrack installation, so I think you might really want to be doing:

rm -f /opt/mtrack/var/cmdcache/*

--Wez.

[Wez Furlong]

On Apr 29, 2012, at 11:28 PM, Sergio A. Kessler wrote:

> ok, I fixed the issue changing htmlspecialchars() by htmlentities()
>
> so, the line:
> $line = htmlspecialchars($line);
> becomes:
> $line = htmlentities($line);
>
> and now, the diff is displaying just rigth... ;-)
>
> (thanks to w3c that almost all iso88591 have html entities)
>
> but this is not the approach of Trac, because the source page of trac
> doesn't have the accented characters as html entities, they show it as
> utf8...

It sounds like there needs to be some kind of encoding configuration or auto-detection for this sort of thing. I'm generally in favor of a local configuration directive (settable in config.ini) that we'll use to transcode source files when displaying them in HTML.

--Wez.

[Sergio A. Kessler]

ho !
I was ommiting the 'var/' !!, f* me!

[Sergio A. Kessler]

yup, maybe something like repo_charset in config.ini

but, meanwhile, I think htmlentities() is better than htmlspecialchars(),
at least in a big part of the world... ;-)


thanks,
/sak