1x svn with 2x Tracs, how to filter commit messages?

[Cooke, Mark]

Folks,

I have one svn repository that is used by two Trac environments (the second created after a major re-write). Unfortunately there is some duplication of ticket numbers and I want to make sure that the commit message gets to the correct ticket.

I found this thread:-

https://groups.google.com/forum/#!topic/trac-users/JmePlTBHklM

...but it is not clear to me how to solve my problem.

A solution that occurs to me is to use different 'envelope' characters for each Trac (e.g. "[]" or "<>") so that only one of the two tracs picks up the message, then set my post-commit hook to call both:

@start C:\Python26\Scripts\trac-admin.exe D:\trac\env1 changeset added "%1" "%2"
@start C:\Python26\Scripts\trac-admin.exe D:\trac\env2 changeset added "%1" "%2"

(note I am on windoze here!)

Can anyone suggest a better solution?

Many thanks,

~ Mark C

[RjOllos]

Using an envelope is an interesting idea. Thinking similarly, you could define  commit_ticket_update_commands.close and commit_ticket_update_commands.refs uniquely for each environment. For example, instead of the token "refs", you could define the token "refs:proj1" or "proj1:refs".

There is also:

https://trac-hacks.org/wiki/MultiProjectCommitTicketUpdaterPlugin

Another possibility is to set the starting ticket number for one of the environments to a large-enough value that the sequences will never overlap.

I'll be interested to hear what solution you end up going with. I could see Trac doing something as an incremental step towards a multiproject environment.

- Ryan

[Cooke, Mark]

> -----Original Message-----
> From: RjOllos [mailto:rjo...@gmail.com]
> Sent: 10 May 2016 20:52
>
> On Tuesday, May 10, 2016 at 6:59:35 AM UTC-7, Cooke, Mark wrote:
>
> > Folks,
> >
> > I have one svn repository that is used by two Trac environments (the
> > second created after a major re-write). Unfortunately there is some
> > duplication of ticket numbers and I want to make sure that the
> > commit message gets to the correct ticket.
> >
> > I found this thread:-
> >
> > https://groups.google.com/forum/#!topic/trac-users/JmePlTBHklM
> >
> > ...but it is not clear to me how to solve my problem.
> >
> > A solution that occurs to me is to use different 'envelope'
> > characters for each Trac (e.g. "[]" or "<>") so that only one of the
> > two tracs picks up the message, then set my post-commit hook to call
> > both:
> >

> > @start trac-admin.exe D:\trac\env1 changeset added "%1" "%2"
> > @start trac-admin.exe D:\trac\env2 changeset added "%1" "%2"

> >
> > (note I am on windoze here!)
> >
> > Can anyone suggest a better solution?
> >
> > Many thanks,
> >
> > ~ Mark C
> >
>
> Using an envelope is an interesting idea. Thinking similarly, you could
> define commit_ticket_update_commands.close and
> commit_ticket_update_commands.refs uniquely for each environment. For
> example, instead of the token "refs", you could define the token "refs:proj1"
> or "proj1:refs".

Hmm, interesting idea but we do not (yet) use the commands, only the references (default set to `<ALL>`).

> There is also:
> https://trac-hacks.org/wiki/MultiProjectCommitTicketUpdaterPlugin

Thanks! That looks ideal (if a bit wordy!), I will have a look at the code. Using existing inter-Trac prefixes is consistent with "normal" usage

> Another possibility is to set the starting ticket number for one of the
> environments to a large-enough value that the sequences will never overlap.

I did not know you could do that (learn a new thing every day)

> I'll be interested to hear what solution you end up going with. I could see
> Trac doing something as an incremental step towards a multiproject
> environment.

At the moment we are just sending all commits to the new project but that is not ideal.

I had another thought to replace the initial svn hook with a python script that can then examine the message to identify the project (e.g. by keyword) which would negate the need to call trac-admin twice. However that is more flaky as svn hooks are not backed up by default `hotcopy` and hides the solution outside of Trac...

> - Ryan

[RjOllos]

SVN 1.8 adds the ability to store the authz file in the/a repository. I'm hoping it's also eventually possible to store the hook scripts in the/a repository, but I haven't looked to see if that's on the radar of the Subversion development team.

https://subversion.apache.org/docs/release-notes/1.8.html#in-repo-authz

Off-topic, but another really nice SVN 1.8 feature is the ability to configure the hook scripts environment through a config file, which can be shared among repositories.

https://subversion.apache.org/docs/release-notes/1.8.html#hooks-env

- Ryan

[Miguel Almeida]

I revisit this thread to see if you can help me setting up a solution for this.

I'm going the MultiProjectCommitTicketUpdaterPlugin route. I've set it up like [1].
Works:
- a commit "foo (see xxx:#99) ", adds reference to ticket 99 in project xxx
Doesn't work:
1.  a commit "foo (closes xxx:#99) ", does not reference nor close  ticket 99 in project xxx

I've also tried:
- removing the configuration commit_ticket_update in [trac] (see [2])
- removing the configuration commit_ticket_update and adding just: commit_ticket_update_check_perms = false


Do you have any idea why tickets are not being closed?

[1] Configuration in  global_trac.ini

[components]
multicommitupdater.* = enabled

[multicommitupdater]
envelope = ()
commands.refs = re see rel

commands.close = fix fixes fixed-forever close closes

[2] Original settings in [ticket] section
commit_ticket_update_check_perms = false
commit_ticket_update_commands.close = close closed closes fix fixed fixes
commit_ticket_update_commands.refs = <ALL>
commit_ticket_update_envelope = ()
commit_ticket_update_notify = true

[Miguel Almeida]

[Update]

Enabled trac debugging, and when committing with (closes project:#137) I get an error in the log [3].
Am trying to figure out why this error happens and how to solve it.

[3]
2016-07-08 19:41:26,827 Trac[commitupdater] ERROR: Unexpected error while processing ticket #137: ExtractionError: Can't extract file(s) to egg cache

The following error occurred while trying to extract file(s) to the Python egg
cache:

  [Errno 13] Permission denied: '/home/svn/.python-eggs'

The Python egg cache directory is currently set to:

  /home/svn/.python-eggs

Perhaps your account does not have write access to this directory?  You can
change the cache directory by setting the PYTHON_EGG_CACHE environment
variable to point to an accessible directory.

[RjOllos]

On Friday, July 8, 2016 at 11:44:16 AM UTC-7, Miguel Almeida wrote:

[Update]

Enabled trac debugging, and when committing with (closes project:#137) I get an error in the log [3].
Am trying to figure out why this error happens and how to solve it.

[3]
2016-07-08 19:41:26,827 Trac[commitupdater] ERROR: Unexpected error while processing ticket #137: ExtractionError: Can't extract file(s) to egg cache

The following error occurred while trying to extract file(s) to the Python egg
cache:

  [Errno 13] Permission denied: '/home/svn/.python-eggs'

The Python egg cache directory is currently set to:

  /home/svn/.python-eggs

Perhaps your account does not have write access to this directory?  You can
change the cache directory by setting the PYTHON_EGG_CACHE environment
variable to point to an accessible directory.

Are you running TracStandalone, or a different web server?

https://trac.edgewall.org/wiki/TracInstall#SettingupthePluginCache

See also:
https://trac.edgewall.org/wiki/TracPlugins#Withan.eggfile

https://trac.edgewall.org/wiki/TracModPython#PythonEggCache

https://trac.edgewall.org/wiki/TracModWSGI#Averybasicscript

- Ryan

[RjOllos]

On Friday, July 8, 2016 at 11:22:00 AM UTC-7, Miguel Almeida wrote:

I revisit this thread to see if you can help me setting up a solution for this.

I'm going the MultiProjectCommitTicketUpdaterPlugin route. I've set it up like [1].
Works:
- a commit "foo (see xxx:#99) ", adds reference to ticket 99 in project xxx
Doesn't work:
1.  a commit "foo (closes xxx:#99) ", does not reference nor close  ticket 99 in project xxx

I've also tried:
- removing the configuration commit_ticket_update in [trac] (see [2])
- removing the configuration commit_ticket_update and adding just: commit_ticket_update_check_perms = false


Do you have any idea why tickets are not being closed?

The user must have TICKET_MODIFY in order to close a ticket, assuming a typical workflow. Do you share an authentication store between Trac and Subversion? There can sometimes also be issues with case-sensitive usernames:

https://trac.edgewall.org/wiki/TracIni#/ignore_auth_case

[Miguel Almeida]

Hi Ryan,

Only now did I have the chance to review this.

I have Trac running in Apache. In apache's trac configuration file the PYTHON_EGG_CACHE is defined:

 <Location /projects>
                SetHandler mod_python
                PythonInterpreter main_interpreter
                PythonHandler trac.web.modpython_frontend
                PythonOption TracEnvParentDir /srv/trac
                PythonOption TracUriRoot /projects
                PythonOption TracLocale en_GB.UTF8
                PythonOption PYTHON_EGG_CACHE /srv/trac/egg-cache
        </Location>

However, this doesn't seem to be respected, as a commit creates the log [4]. Notice the error says The Python egg cache directory is currently set to: /home/svn/.python-egg.

As for "The user must have TICKET_MODIFY" part: authentications are completely separate, even though usernames in both places are the same. All lowercase. Is this how permission is checked (ie, only by the user's string, even though the authentications are in different places)? Nevertheless, I believe the setting "commit_ticket_update_check_perms = false" should ignore permissions, right?

Since the exception's message is consistent with the code blcok where the ticket should be updated I suspect this permission error is causing the ticket not to update. I'm still unsure as to why this is happening though.

[4] Trac log, even with PythonOption PYTHON_EGG_CACHE /srv/trac/egg-cache

2016-07-14 18:21:29,830 Trac[api] DEBUG: Event changeset_added on IWRS for revision 6014
2016-07-14 18:21:29,834 Trac[commitupdater] DEBUG: Updating ticket #137
2016-07-14 18:21:29,835 Trac[default_workflow] DEBUG: Workflow actions at initialization: {u'resolve': {u'operations': [u'set_resolution'], 'name': u'resolve', 'default': 0, 'newstate': u'closed', 'oldstates': [u'new', u'assigned', u'accepted', u'reopened'], u'permissions': [u'TICKET_MODIFY']}, u'accept': {u'operations': [u'set_owner_to_self'], 'name': u'accept', 'default': 0, 'newstate': u'accepted', 'oldstates': [u'new', u'assigned', u'accepted', u'reopened'], u'permissions': [u'TICKET_MODIFY']}, u'leave': {u'operations': [u'leave_status'], 'name': u'leave', u'default': 1, 'newstate': u'*', 'oldstates': [u'*'], 'permissions': []}, u'reopen': {u'operations': [u'del_resolution'], 'name': u'reopen', 'default': 0, 'newstate': u'reopened', 'oldstates': [u'closed'], u'permissions': [u'TICKET_CREATE']}, u'reassign': {u'operations': [u'set_owner'], 'name': u'reassign', 'default': 0, 'newstate': u'assigned', 'oldstates': [u'new', u'assigned', u'accepted', u'reopened'], u'permissions': [u'TICKET_MODIFY']}, '_reset': {'operations': ['reset_workflow'], 'name': 'reset', 'default': 0, 'newstate': 'new', 'oldstates': [], 'permissions': []}}

2016-07-14 18:21:29,835 Trac[api] DEBUG: action controllers for ticket workflow: ['ConfigurableTicketWorkflow']
2016-07-14 18:21:29,879 Trac[commitupdater] ERROR: Unexpected error while processing ticket #137: ExtractionError: Can't extract file(s) to egg cache

The following error occurred while trying to extract file(s) to the Python egg
cache:

  [Errno 13] Permission denied: '/home/svn/.python-eggs'

The Python egg cache directory is currently set to:

  /home/svn/.python-eggs

Perhaps your account does not have write access to this directory?  You can
change the cache directory by setting the PYTHON_EGG_CACHE environment
variable to point to an accessible directory.

--
You received this message because you are subscribed to a topic in the Google Groups "Trac Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/trac-users/dOkdYT4nRK4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to trac-users+...@googlegroups.com.
To post to this group, send email to trac-...@googlegroups.com.
Visit this group at https://groups.google.com/group/trac-users.
For more options, visit https://groups.google.com/d/optout.

[RjOllos]

On Thursday, July 14, 2016 at 10:55:00 AM UTC-7, Miguel Almeida wrote:

Hi Ryan,

Only now did I have the chance to review this.

I have Trac running in Apache. In apache's trac configuration file the PYTHON_EGG_CACHE is defined:

 <Location /projects>
                SetHandler mod_python
                PythonInterpreter main_interpreter
                PythonHandler trac.web.modpython_frontend
                PythonOption TracEnvParentDir /srv/trac
                PythonOption TracUriRoot /projects
                PythonOption TracLocale en_GB.UTF8
                PythonOption PYTHON_EGG_CACHE /srv/trac/egg-cache
        </Location>

However, this doesn't seem to be respected, as a commit creates the log [4]. Notice the error says The Python egg cache directory is currently set to: /home/svn/.python-egg.

I misled you earlier. You'll need to set the environment variable PYTHON_EGG_CACHE in your post-commit hook script, as described here:

https://trac.edgewall.org/wiki/TracRepositoryAdmin#ExplicitSync

 

As for "The user must have TICKET_MODIFY" part: authentications are completely separate, even though usernames in both places are the same. All lowercase. Is this how permission is checked (ie, only by the user's string, even though the authentications are in different places)? Nevertheless, I believe the setting "commit_ticket_update_check_perms = false" should ignore permissions, right?

Since the exception's message is consistent with the code blcok where the ticket should be updated I suspect this permission error is causing the ticket not to update. I'm still unsure as to why this is happening though.

It looks like MultiProjectCommitTicketUpdater has a different option. You could try:

[multicommitupdater]

check_perms = false

https://trac-hacks.org/browser/multiprojectcommitticketupdaterplugin/0.12/multicommitupdater/commitupdater.py?marks=167&rev=11453#L147

- Ryan

[Miguel Almeida]

Hi,

On Thu, Jul 14, 2016 at 8:11 PM, RjOllos <rjo...@gmail.com> wrote:

On Thursday, July 14, 2016 at 10:55:00 AM UTC-7, Miguel Almeida wrote:

Hi Ryan,

Only now did I have the chance to review this.

I have Trac running in Apache. In apache's trac configuration file the PYTHON_EGG_CACHE is defined:

 <Location /projects>
                SetHandler mod_python
                PythonInterpreter main_interpreter
                PythonHandler trac.web.modpython_frontend
                PythonOption TracEnvParentDir /srv/trac
                PythonOption TracUriRoot /projects
                PythonOption TracLocale en_GB.UTF8
                PythonOption PYTHON_EGG_CACHE /srv/trac/egg-cache
        </Location>

However, this doesn't seem to be respected, as a commit creates the log [4]. Notice the error says The Python egg cache directory is currently set to: /home/svn/.python-egg.

I misled you earlier. You'll need to set the environment variable PYTHON_EGG_CACHE in your post-commit hook script, as described here:

https://trac.edgewall.org/wiki/TracRepositoryAdmin#ExplicitSync

 This is actually already happening:

#Trac notification
export PYTHON_EGG_CACHE="/srv/trac/egg-cache/"
echo "repos: " $REPOS >>  /srv/svn/test
echo "rev: " $REV >>  /srv/svn/test
sudo -u www-data trac-admin /srv/trac/iwrs/ changeset added $REPOS $REV
sudo -u www-data trac-admin /srv/trac/tools/ changeset added $REPOS $REV
sudo -u www-data trac-admin /srv/trac/bibliovigilance/ changeset added $REPOS $REV

But the log is still saying the Python egg cache directory is currently set to: /home/svn/.python-eggs

This is definitely related with the users running these commands.
- From the error log, the user that is seen is svn (so it is searching for the cache in /home/svn/...
- svn is running via /usr/bin/svnserve, which is a process

- However, as seen above trac-admin should be running with user www-data

- I've also tried adding sudo -E (to preserve the environment variables), but apparently this can't be done here (sudo: sorry, you are not allowed to preserve the environment)



Still searching for the solution.


 

As for "The user must have TICKET_MODIFY" part: authentications are completely separate, even though usernames in both places are the same. All lowercase. Is this how permission is checked (ie, only by the user's string, even though the authentications are in different places)? Nevertheless, I believe the setting "commit_ticket_update_check_perms = false" should ignore permissions, right?

Since the exception's message is consistent with the code blcok where the ticket should be updated I suspect this permission error is causing the ticket not to update. I'm still unsure as to why this is happening though.

It looks like MultiProjectCommitTicketUpdater has a different option. You could try:

[multicommitupdater]

check_perms = false

https://trac-hacks.org/browser/multiprojectcommitticketupdaterplugin/0.12/multicommitupdater/commitupdater.py?marks=167&rev=11453#L147

Thanks for the tip. I've updated the tool's documentation to indicate this (https://trac-hacks.org/wiki/MultiProjectCommitTicketUpdaterPlugin#Options)

[Jun Omae]

On Fri, Jul 15, 2016 at 6:55 PM, Miguel Almeida
<migueld...@gmail.com> wrote:
> #Trac notification
> export PYTHON_EGG_CACHE="/srv/trac/egg-cache/"
> echo "repos: " $REPOS >> /srv/svn/test
> echo "rev: " $REV >> /srv/svn/test
> sudo -u www-data trac-admin /srv/trac/iwrs/ changeset added $REPOS $REV
> sudo -u www-data trac-admin /srv/trac/tools/ changeset added $REPOS $REV
> sudo -u www-data trac-admin /srv/trac/bibliovigilance/ changeset added
> $REPOS $REV
>
> But the log is still saying the Python egg cache directory is currently set
> to: /home/svn/.python-eggs
>
> This is definitely related with the users running these commands.
> - From the error log, the user that is seen is svn (so it is searching for
> the cache in /home/svn/...
> - svn is running via /usr/bin/svnserve, which is a process
> - However, as seen above trac-admin should be running with user www-data
> - I've also tried adding sudo -E (to preserve the environment variables),
> but apparently this can't be done here (sudo: sorry, you are not allowed to
> preserve the environment)
>
> Still searching for the solution.

env_reset option for sudo is enabled by default. The PYTHON_EGG_CACHE
variable doesn't pass to trac-admin via sudo. Instead, try env
PYTHON_EGG_CACHE="..." like this:

---->8---->8---->8---->8---->8---->8---->8----

echo "repos: " $REPOS >> /srv/svn/test
echo "rev: " $REV >> /srv/svn/test

sudo -u www-data env PYTHON_EGG_CACHE="/srv/trac/egg-cache/" /bin/sh <<EOF

trac-admin /srv/trac/iwrs/ changeset added $REPOS $REV

trac-admin /srv/trac/tools/ changeset added $REPOS $REV

trac-admin /srv/trac/bibliovigilance/ changeset added $REPOS $REV

EOF
---->8---->8---->8---->8---->8---->8---->8----


--
Jun Omae <jun...@gmail.com> (大前 潤)

[Jun Omae]

On Fri, Jul 15, 2016 at 7:05 PM, Jun Omae <jun...@gmail.com> wrote:
> env_reset option for sudo is enabled by default. The PYTHON_EGG_CACHE
> variable doesn't pass to trac-admin via sudo. Instead, try env
> PYTHON_EGG_CACHE="..." like this:
>
> ---->8---->8---->8---->8---->8---->8---->8----
> echo "repos: " $REPOS >> /srv/svn/test
> echo "rev: " $REV >> /srv/svn/test
> sudo -u www-data env PYTHON_EGG_CACHE="/srv/trac/egg-cache/" /bin/sh <<EOF
> trac-admin /srv/trac/iwrs/ changeset added $REPOS $REV
> trac-admin /srv/trac/tools/ changeset added $REPOS $REV
> trac-admin /srv/trac/bibliovigilance/ changeset added $REPOS $REV
> EOF
> ---->8---->8---->8---->8---->8---->8---->8----

Well, the env command isn't needed. The following is simpler than.

echo "repos: " $REPOS >> /srv/svn/test
echo "rev: " $REV >> /srv/svn/test

sudo -u www-data /bin/sh <<EOF
export PYTHON_EGG_CACHE="/srv/trac/egg-cache/"

trac-admin /srv/trac/iwrs/ changeset added $REPOS $REV
trac-admin /srv/trac/tools/ changeset added $REPOS $REV
trac-admin /srv/trac/bibliovigilance/ changeset added $REPOS $REV
EOF

[Miguel Almeida]

I think this is definitely what I want, although now I'm getting a "sudo no tty present and no askpass program specified" which, for some reason, I wasn't getting when I ran the sudo commands one by one instead of within he EOF!

 

--
Jun Omae <jun...@gmail.com> (大前 潤)

[Miguel Almeida]

[UPDATE]

The "sudo no tty present and no askpass program specified" problem was caused by the fact that I'd changed "sudo -u www-data trac-admin" to "sudo -u www-data /bin/sh".
My sudoers file was configured so that the svn user could only run trac-admin as www-data (so he couldn't run /bin/sh).

To resolve the problem in a secured fashion, I ended up with:
export PYTHON_EGG_CACHE="/srv/trac/egg-cache/"
sudo -u www-data -E trac-admin /srv/trac/tools/ changeset added $REPOS $REV
(...)

i.e, I passed -E
And I added :SETENV: to the sudoers' configuration, so that svn can use SETENV when running trac-admin as www-data.


Everything now works and all is well in the world. Especially with Portugal as European Champions!

Thank you all for the help!
Miguel