[TracTicketChangeLogPlugin] Failiing to display tickets with long revision messages

Riedel, Torge

unread,

Dec 21, 2015, 9:37:36 AM12/21/15

to trac-...@googlegroups.com

Hi,

today I faced a problem with TracTicketChangeLogPlugin:

After analyzing the code and adding some debug outputs I can say the following:

1. Ticket number is #39

2. The db-query gives 46 revisions where LIKE '%#39%' matches (which is much more than really necessary, since revisions referencing tickets e.g. #391 are returned too)

3. The reg-ex matching hangs when there is a long commit message (sorry I cannot post it here since it contains confidential information)

a. The message contains Unicode chars, but changing the encoding of the message to match to ‘ascii’ with ‘ignore’ option does not help

b. The message has 41 lines (it’s a message of a tag commit in subversion, where we list all tickets solved in this version, the format is:
Version vX.Y
----
* #n1 Fixed this
* #n2 Fixed that
… and so on

c. The whole message has a count of 3073 chars

The log pattern is configured to:

[ticketlog]

log_pattern = (\n|.)*#%s($|\D(\n|.)*)

Which gives very good results. In general. Since I’m not a python developer, help is appreciated.

Regards

Torge

CENIT AG, Industriestrasse 52-54, 70565 Stuttgart, Tel.: +49 711 7825-30, Fax: +49 711 7825-4000, Internet: www.cenit.com
Geschaeftsstellen: Berlin, Frankfurt, Hamburg, Hannover, Muenchen, Oelsnitz, Ratingen, Saarbruecken
Vorstandsmitglieder: Kurt Bengel, Matthias Schmidt
Aufsichtsratsmitglieder: Andreas Schmidt (Vorsitzender des Aufsichtsrats), Hubert Leypoldt, Andreas Karrer
Bankverbindungen:
Deutsche Bank (BLZ 600 700 70) Kto. 1661 040 IBAN : DE85 6007 0070 0166 1040 00 SWIFT-CODE : DEUTDESS,
Commerzbank (BLZ 600 400 71) Kto. 532 015 500 IBAN : DE83 6004 0071 0532 0155 00 SWIFT-Code : COBADEFF600,
Registergericht: Amtsgericht Stuttgart
Handelsregister: HRB Nr. 19117
Umsatzsteuer: ID-Nr. DE 147 862 777

RjOllos

unread,

Dec 22, 2015, 1:07:16 PM12/22/15

to Trac Users, T.Ri...@cenit.de

On Monday, December 21, 2015 at 6:37:36 AM UTC-8, Riedel, Torge wrote:

Hi,

today I faced a problem with TracTicketChangeLogPlugin:

After analyzing the code and adding some debug outputs I can say the following:

1.       Ticket number is #39

2.       The db-query gives 46 revisions where LIKE '%#39%' matches (which is much more than really necessary, since revisions referencing tickets e.g. #391 are returned too)

3.       The reg-ex matching hangs when there is a long commit message (sorry I cannot post it here since it contains confidential information)

a.       The message contains Unicode chars, but changing the encoding of the message to match to ‘ascii’ with ‘ignore’ option does not help

b.      The message has 41 lines (it’s a message of a tag commit in subversion, where we list all tickets solved in this version, the format is:
Version vX.Y
----
* #n1 Fixed this
* #n2 Fixed that
… and so on

c.       The whole message has a count of 3073 chars

The log pattern is configured to:

[ticketlog]

log_pattern = (\n|.)*#%s($|\D(\n|.)*)

Which gives very good results. In general. Since I’m not a python developer, help is appreciated.

Regards

Torge

You could try the patterns mentioned in this ticket:

https://trac-hacks.org/ticket/7884

Torge Riedel

unread,

Dec 23, 2015, 6:42:43 AM12/23/15

to trac-...@googlegroups.com

Am 22.12.2015 um 19:07 schrieb RjOllos:
>
>
> On Monday, December 21, 2015 at 6:37:36 AM UTC-8, Riedel, Torge wrote:
>
> Hi,
>
> today I faced a problem with TracTicketChangeLogPlugin:
>
> After analyzing the code and adding some debug outputs I can say the following:
>
> 1.Ticket number is #39
>

> 2.The db-query gives 46 revisions where LIKE '%#39%' matches (which is much more than really necessary, since revisions referencing tickets e.g. #391 are returned too)
>
> 3.The reg-ex matching hangs when there is a long commit message (sorry I cannot post it here since it contains confidential information)
>
> a.The message contains Unicode chars, but changing the encoding of the message to match to ‘ascii’ with ‘ignore’ option does not help
>
> b.The message has 41 lines (it’s a message of a tag commit in subversion, where we list all tickets solved in this version, the format is:

> Version vX.Y
> ----
> * #n1 Fixed this
> * #n2 Fixed that
> … and so on
>

> c.The whole message has a count of 3073 chars

>
> The log pattern is configured to:
>
> [ticketlog]
>
> log_pattern = (\n|.)*#%s($|\D(\n|.)*)
>
> Which gives very good results. In general. Since I’m not a python developer, help is appreciated.
>
> Regards
>
> Torge
>
>
> You could try the patterns mentioned in this ticket:
> https://trac-hacks.org/ticket/7884
> See also:
> https://trac-hacks.org/ticket/12436
>
> I suspect the issue you describe as "hanging on a long commit message" is due to searching every log message on every request. I hope to fix that in #11821, but I don't have a timetable yet for implementing the changes.
> https://trac-hacks.org/ticket/11821
>
> - Ryan
>

> --
> You received this message because you are subscribed to the Google Groups "Trac Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to trac-users+...@googlegroups.com <mailto:trac-users+...@googlegroups.com>.
> To post to this group, send email to trac-...@googlegroups.com <mailto:trac-...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/trac-users.
> For more options, visit https://groups.google.com/d/optout.

Hi Ryan,

well my issue is not really due to the number of log messages. There is a pre-selection of the log messages by a SELECT-query using a "LIKE" on the message. I added debug prints in the plug-in, activated the log to the file in the trac instance and called "tail -f trac.log" to see what happens when visiting the ticket page. And I see that the long message (it is the 11th of 46 log messages to check) is passed to the re.match()-method and then nothing more is logged. Apache is no more responding to other requests.
Maybe this is a bug in python regex code? Can you help me to check this? I'm not a python developer so I need help. Everything more complex than adding debug prints is out of my possibilities! ;-) I think of a small python script where I can pass the ticket id ("python test.py 39") executed in the directory where the trac.db is located. This script is running the code snippet of the plug-in with log of debug prints. So I can check whether this is a python issue and maybe it's possible for me to reduce it to non-confidential data and open an issue.

Thanks
Torge

RjOllos

unread,

Dec 23, 2015, 12:39:06 PM12/23/15

to Trac Users, torge...@gmx.de

> To unsubscribe from this group and stop receiving emails from it, send an email to trac-users+unsubscribe@googlegroups.com <mailto:trac-users+unsub...@googlegroups.com>.
> To post to this group, send email to trac-...@googlegroups.com <mailto:trac-users@googlegroups.com>.

> Visit this group at https://groups.google.com/group/trac-users.
> For more options, visit https://groups.google.com/d/optout.

Hi Ryan,

well my issue is not really due to the number of log messages. There is a pre-selection of the log messages by a SELECT-query using a "LIKE" on the message. I added debug prints in the plug-in, activated the log to the file in the trac instance and called "tail -f trac.log" to see what happens when visiting the ticket page. And I see that the long message (it is the 11th of 46 log messages to check) is passed to the re.match()-method and then nothing more is logged. Apache is no more responding to other requests.
Maybe this is a bug in python regex code? Can you help me to check this? I'm not a python developer so I need help. Everything more complex than adding debug prints is out of my possibilities! ;-) I think of a small python script where I can pass the ticket id ("python test.py 39") executed in the directory where the trac.db is located. This script is running the code snippet of the plug-in with log of debug prints. So I can check whether this is a python issue and maybe it's possible for me to reduce it to non-confidential data and open an issue.

Thanks
Torge

#11821 also proposes to improve the regex. I think we should only extract tokens like "Refs #1", rather than just "#1", like CommitTicketUpdater does. Approximately how many tickets are referenced in the log message?

- Ryan

RjOllos

unread,

Dec 23, 2015, 8:13:03 PM12/23/15

to Trac Users, torge...@gmx.de

Are your ticket references of the form #\d+, or do you prefix the ticket ID references in your log messages #\d+ with a CommitTicketUpdater command?

http://trac.edgewall.org/wiki/CommitTicketUpdater#Configure

- Ryan

Torge Riedel

unread,

Dec 25, 2015, 4:54:43 AM12/25/15

to RjOllos, Trac Users

> > https://trac-hacks.org/ticket/7884 <https://trac-hacks.org/ticket/7884>
> > See also:
> > https://trac-hacks.org/ticket/12436 <https://trac-hacks.org/ticket/12436>

> >
> > I suspect the issue you describe as "hanging on a long commit message" is due to searching every log message on every request. I hope to fix that in #11821, but I don't have a timetable yet for implementing the changes.

> > https://trac-hacks.org/ticket/11821 <https://trac-hacks.org/ticket/11821>

> >
> > - Ryan
> >
> > --
> > You received this message because you are subscribed to the Google Groups "Trac Users" group.

>
> Hi Ryan,
>
> well my issue is not really due to the number of log messages. There is a pre-selection of the log messages by a SELECT-query using a "LIKE" on the message. I added debug prints in the plug-in, activated the log to the file in the trac instance and called "tail -f trac.log" to see what happens when visiting the ticket page. And I see that the long message (it is the 11th of 46 log messages to check) is passed to the re.match()-method and then nothing more is logged. Apache is no more responding to other requests.
> Maybe this is a bug in python regex code? Can you help me to check this? I'm not a python developer so I need help. Everything more complex than adding debug prints is out of my possibilities! ;-) I think of a small python script where I can pass the ticket id ("python test.py 39") executed in the directory where the trac.db is located. This script is running the code snippet of the plug-in with log of debug prints. So I can check whether this is a python issue and maybe it's possible for me to reduce it to non-confidential data and open an issue.
>
> Thanks
> Torge
>
>
> #11821 also proposes to improve the regex. I think we should only extract tokens like "Refs #1", rather than just "#1", like CommitTicketUpdater does. Approximately how many tickets are referenced in the log message?
>
> - Ryan
>
>
> Are your ticket references of the form #\d+, or do you prefix the ticket ID references in your log messages #\d+ with a CommitTicketUpdater command?
> http://trac.edgewall.org/wiki/CommitTicketUpdater#Configure
>
> - Ryan

Well, it's a tag commit where we always list all fixed issues. The message has 41 lines, two lines for version and horizontal delimiter -> 39 ticket references. Our format is always like this:

Version vX.Y
----
* #n1 Fixed this
* #n2 Fixed that

...

Where "..." stands for more list points. We do not use CommitTicketUpdater.
Again: It looks to me, that regex engine of python is hanging when trying to find a match for (\n|.)*#39($|\D(\n|.)*) in the long message. I like to verify this using a small python script, but I'm not a python developer.

Improving the regex is only one solution and - in my case - not suitable if the reference format changes. This would skip all existing commits ins svn, where the new format is not used.

BTW: Merry christmas!

Regards
Torge

RjOllos

unread,

Dec 28, 2015, 4:41:21 PM12/28/15

to Trac Users, rjo...@gmail.com, torge...@gmx.de

> > To unsubscribe from this group and stop receiving emails from it, send an email to trac-users+unsubscribe@googlegroups.com <mailto:trac-users%2Bunsu...@googlegroups.com> <mailto:trac-users+unsub...@googlegroups.com <mailto:trac-users%2Bunsu...@googlegroups.com>>.
> > To post to this group, send email to trac-...@googlegroups.com <mailto:trac-users@googlegroups.com> <mailto:trac-users@googlegroups.com <mailto:trac-users@googlegroups.com>>.

> > Visit this group at https://groups.google.com/group/trac-users <https://groups.google.com/group/trac-users>.
> > For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
> Hi Ryan,
>
> well my issue is not really due to the number of log messages. There is a pre-selection of the log messages by a SELECT-query using a "LIKE" on the message. I added debug prints in the plug-in, activated the log to the file in the trac instance and called "tail -f trac.log" to see what happens when visiting the ticket page. And I see that the long message (it is the 11th of 46 log messages to check) is passed to the re.match()-method and then nothing more is logged. Apache is no more responding to other requests.
> Maybe this is a bug in python regex code? Can you help me to check this? I'm not a python developer so I need help. Everything more complex than adding debug prints is out of my possibilities! ;-) I think of a small python script where I can pass the ticket id ("python test.py 39") executed in the directory where the trac.db is located. This script is running the code snippet of the plug-in with log of debug prints. So I can check whether this is a python issue and maybe it's possible for me to reduce it to non-confidential data and open an issue.
>
> Thanks
> Torge
>
>
> #11821 also proposes to improve the regex. I think we should only extract tokens like "Refs #1", rather than just "#1", like CommitTicketUpdater does. Approximately how many tickets are referenced in the log message?
>
> - Ryan
>
>
> Are your ticket references of the form #\d+, or do you prefix the ticket ID references in your log messages #\d+ with a CommitTicketUpdater command?
> http://trac.edgewall.org/wiki/CommitTicketUpdater#Configure
>
> - Ryan

Well, it's a tag commit where we always list all fixed issues. The message has 41 lines, two lines for version and horizontal delimiter -> 39 ticket references. Our format is always like this:

Version vX.Y
----
* #n1 Fixed this
* #n2 Fixed that
...

Where "..." stands for more list points. We do not use CommitTicketUpdater.
Again: It looks to me, that regex engine of python is hanging when trying to find a match for (\n|.)*#39($|\D(\n|.)*) in the long message. I like to verify this using a small python script, but I'm not a python developer.

Improving the regex is only one solution and - in my case - not suitable if the reference format changes. This would skip all existing commits ins svn, where the new format is not used.

BTW: Merry christmas!

Regards
Torge

Thank you for the additional details. I added a note to #12621 to investigate. I'll try to make those changes backward compatible. I think it would be best to continue the discussion in ticket #12621, so you can receive notifications of changesets.

- Ryan

Riedel, Torge

unread,

Jan 4, 2016, 7:17:28 AM1/4/16

to trac-...@googlegroups.com

Hi Ryan,

after reading through all related stuff I’ve seen this comment https://trac-hacks.org/ticket/7884#comment:5 where someone replied to my comment:4 and tells about hanging server. After using the regexp proposed in comment:5 this is fixed.

Sorry, that I didn’t see that earlier.

I added this pattern on page https://trac-hacks.org/wiki/TracTicketChangelogPlugin with revision 23 so others can find this faster.

Regards

Torge

> > To unsubscribe from this group and stop receiving emails from it, send an email to trac-users+...@googlegroups.com <mailto:trac-users%2Bunsu...@googlegroups.com> <mailto:trac-users+...@googlegroups.com <mailto:trac-users%2Bunsu...@googlegroups.com>>.
> > To post to this group, send email to trac-...@googlegroups.com <mailto:trac-...@googlegroups.com> <mailto:trac-...@googlegroups.com <mailto:trac-...@googlegroups.com>>.

--

You received this message because you are subscribed to the Google Groups "Trac Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trac-users+...@googlegroups.com.
To post to this group, send email to trac-...@googlegroups.com.
Visit this group at https://groups.google.com/group/trac-users.
For more options, visit https://groups.google.com/d/optout.

RjOllos

unread,

Apr 4, 2017, 4:11:03 AM4/4/17

to Trac Users, T.Ri...@cenit.de

On Monday, January 4, 2016 at 4:17:28 AM UTC-8, Riedel, Torge wrote:

Hi Ryan,

after reading through all related stuff I’ve seen this comment https://trac-hacks.org/ticket/7884#comment:5 where someone replied to my comment:4 and tells about hanging server. After using the regexp proposed in comment:5 this is fixed.

Sorry, that I didn’t see that earlier.

I added this pattern on page https://trac-hacks.org/wiki/TracTicketChangelogPlugin with revision 23 so others can find this faster.

Regards

Torge

There's a new version of the plugin that works with Trac 1.2. It creates a table of ticket-revision relations, so the performance should be much better. It also uses CommitTicketUpdater to parse the messages, so the ticket reference syntax will be the same as used in Trac.

http://trac.edgewall.org/wiki/CommitTicketUpdater
https://trac-hacks.org/browser/tracticketchangelogplugin/1.2

- Ryan

RjOllos

unread,

Apr 11, 2017, 12:49:50 AM4/11/17

to Trac Users, T.Ri...@cenit.de

TracBackLinkPlugin is another option that offers similar functionality.

https://trac-hacks.org/wiki/TracBackLinkPlugin

- Ryan

Reply all

Reply to author

Forward