ITicketChangeListener semantics

[Christopher Nelson]

http://www.edgewall.org/docs/branches-0.12-stable/epydoc/trac.ticket.api.ITicketChangeListener-class.html
shows me the syntax/interface for a ticket change listener but doesn't
give me a lot of context for when it is called, what I can safely do
in a listener, etc. I'm running into a problem in my listener that
only shows up on large datasets in our test environment but not in the
small dataset in my development environment and I'd like to understand
timing and context so I know where to look for my problem. Is there
some tutorial or other documentation that will tell me how listeners
interact with the system and with each other? For example, if my
listener saves a ticket, does it get called again recursively so I
have to be reentrant? Does another call get deferred until this call
ends? When do other listeners get called for the change my listener
saves?

Here's what I'm trying to do: when a field related to scheduling
tickets changes, do a minimal schedule recalculation. So, if the
estimate for a ticket change from 8 hours to 16, any following work
will start a day later or this task will start a day earlier. In the
latter case, I'll do all my calculations then have a new start date
for the ticket that the listener got invoked for. When I save that
change -- inside the listener -- what happens?

Chris
--
A: Top-posting.
Q: What is the most annoying thing in e-mail?

[Peter Suter]

On 14.09.2012 15:19, Christopher Nelson wrote:
> Is there
> some tutorial or other documentation that will tell me how listeners
> interact with the system and with each other?

One place where I've been trying to document such things is in [1].
Unfortunately the ITicketChangeListener page is not up yet.

I'm not aware of any other such resource. Unless the Trac source code
counts. :)

> For example, if my
> listener saves a ticket, does it get called again recursively so I
> have to be reentrant? Does another call get deferred until this call
> ends? When do other listeners get called for the change my listener
> saves?
>
> Here's what I'm trying to do: when a field related to scheduling
> tickets changes, do a minimal schedule recalculation. So, if the
> estimate for a ticket change from 8 hours to 16, any following work
> will start a day later or this task will start a day earlier. In the
> latter case, I'll do all my calculations then have a new start date
> for the ticket that the listener got invoked for. When I save that
> change -- inside the listener -- what happens?

All ITicketChangeListener.ticket_changed() implementations get called in
a loop at the end of Ticket.save_changes() [2].

There is no special mechanism to handle reentrancy.
ITicketChangeListener is not meant to be used in situations where the
ticket gets "manipulated" (="changed before saving"). Instead you should
use ITicketManipulator.validate_ticket().

The ITicketManipulator documentation page [3] already exists.
Improvements are of course appreciated.

[1] http://trac.edgewall.org/wiki/TracDev/PluginDevelopment/ExtensionPoints

[2]
http://trac.edgewall.org/browser/tags/trac-1.0/trac/ticket/model.py?marks=362-363#L361

[3]
http://trac.edgewall.org/wiki/TracDev/PluginDevelopment/ExtensionPoints/trac.ticket.api.ITicketManipulator

--
Peter

[Steffen Hoffmann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 14.09.2012 15:19, Christopher Nelson wrote:

> http://www.edgewall.org/docs/branches-0.12-stable/epydoc/trac.ticket.api.ITicketChangeListener-class.html
>
>
shows me the syntax/interface for a ticket change listener but doesn't
> give me a lot of context for when it is called,

A full-text search over Trac sources yields only one place,
trac.ticket.api, where ITicketChangeListener is mentioned (plus one
occurrence in trac.ticket.tests.model - a unit test).

But note:
trac.ticket.api.TicketSystem.

A closer look at trac.ticket.model.Ticket reveals the core logic,that is
executed in appropriate methods there like so:

class Ticket(object):
...

def delete(self, db=None):
...

for listener in TicketSystem(self.env).change_listeners:
listener.ticket_deleted(self)

def get_change(self, cnum=None, cdate=None, db=None):
...

> what I can safely do in a listener, etc. I'm running into a problem
> in my listener that only shows up on large datasets in our test
> environment but not in the small dataset in my development
> environment and I'd like to understand timing and context so I know
> where to look for my problem. Is there some tutorial or other
> documentation that will tell me how listeners interact with the
> system and with each other? For example, if my listener saves a
> ticket, does it get called again recursively so I have to be
> reentrant? Does another call get deferred until this call ends?
> When do other listeners get called for the change my listener saves?

You'll see by now: They're executed right after altering the ticket data
in the db. Straight from that springs the advice to not alter tickets in
ticket change listeners, at least not in such a way, that a change will
trigger another change and that another one and ...

It'll be done ticket by ticket, but by re-spawning you could end up
applying side-effects for a ticket that is actually waiting to get
altered directly too.

> Here's what I'm trying to do: when a field related to scheduling
> tickets changes, do a minimal schedule recalculation. So, if the
> estimate for a ticket change from 8 hours to 16, any following work
> will start a day later or this task will start a day earlier. In
> the latter case, I'll do all my calculations then have a new start
> date for the ticket that the listener got invoked for. When I save
> that change -- inside the listener -- what happens?

I've been reading your questions and explanation already for a while
here. Now I'm forced to disclose some of my personal opinions. Take with
a grain of salt, and listen to others as well.

Doing automatic ticket changes for your PM stuff will not work well, if
at all. Doing regular ticket changes each re-scheduling would leave a
changes entry and change comment for each action to each ticket, and
your tickets history will quickly grow beyond usable limits.

I conclude, that you'll likely need to
* mess directly with PM-related ticket field values and spare the
planning history
* do it in a dedicated table outside of Trac db tables 'ticket' and
'ticket_custom' (preferred).

Sincerely,

Steffen Hoffmann
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAlBTdJAACgkQ31DJeiZFuHee9gCfePgP/faygyF9FeyTtbhOLpEg
5JUAnirJKENqy1QIMtbS8lFjaFq2qDns
=Hn7V
-----END PGP SIGNATURE-----

[Steffen Hoffmann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 14.09.2012 20:16, Steffen Hoffmann wrote:
> But note:
> trac.ticket.api.TicketSystem.

Oh, this was unfinished. I meant to point at this:

change_listeners = ExtensionPoint(ITicketChangeListener)
(in trac.ticket.api.TicketSystem)

Steffen Hoffmann
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAlBTdSsACgkQ31DJeiZFuHfcAwCeOzOyV/9vaVZiWzlKaJkGxK6j
qckAoNMcTqQB3EdreKK9buzXaLtd3i/f
=UL7Y
-----END PGP SIGNATURE-----

[Christopher Nelson]

> On 14.09.2012 15:19, Christopher Nelson wrote:
>> Is there
>> some tutorial or other documentation that will tell me how listeners
>> interact with the system and with each other?
>
> One place where I've been trying to document such things is in [1].
> Unfortunately the ITicketChangeListener page is not up yet.

I appreciate you efforts and have been there a few times. Thanks.

> I'm not aware of any other such resource. Unless the Trac source code
> counts. :)

Of course it does. And I had seen where it gets invoked. But more
broadly, I don't know if Trac gets executed in a single thread (so
only one instance of my listener could be active at a time) or if,
say, each user gets a thread in the server and multiple saves can
fight.

>...

>> Here's what I'm trying to do: when a field related to scheduling
>> tickets changes, do a minimal schedule recalculation. So, if the
>> estimate for a ticket change from 8 hours to 16, any following work
>> will start a day later or this task will start a day earlier. In the
>> latter case, I'll do all my calculations then have a new start date
>> for the ticket that the listener got invoked for. When I save that
>> change -- inside the listener -- what happens?
>
> All ITicketChangeListener.ticket_changed() implementations get called in a
> loop at the end of Ticket.save_changes() [2].

Right. I've seen that.

> There is no special mechanism to handle reentrancy. ITicketChangeListener is
> not meant to be used in situations where the ticket gets "manipulated"
> (="changed before saving"). Instead you should use
> ITicketManipulator.validate_ticket().

Well, I'd argue that I'm not trying to manipulate the ticket as, for
example, I think EstimationTools changes an estimate of "8" to "8.0".
I'm trying to react to changes in some fields to propagate changes to
other tickets (and, fairly often I guess, change additional fields in
the current ticket). You could argue that a manipulator could: 1)
reschedule everything and save all other tickets, 2) set the new
schedule values in the current ticket. But that seems awkward.

> The ITicketManipulator documentation page [3] already exists. Improvements
> are of course appreciated.

>...

That's beautiful. Thank you. I have not comments right now because I
just skimmed it and don't have an immediate need.

[Christopher Nelson]

On Fri, Sep 14, 2012 at 2:16 PM, Steffen Hoffmann <hof...@web.de> wrote:
> On 14.09.2012 15:19, Christopher Nelson wrote:
> A full-text search over Trac sources yields only one place,
> trac.ticket.api, where ITicketChangeListener is mentioned (plus one
> occurrence in trac.ticket.tests.model - a unit test).
>
> But note:
> trac.ticket.api.TicketSystem.
>
> A closer look at trac.ticket.model.Ticket reveals the core logic,that is
> executed in appropriate methods there like so:
>
> class Ticket(object):
> ...
>
> def delete(self, db=None):
> ...
>
> for listener in TicketSystem(self.env).change_listeners:
> listener.ticket_deleted(self)
>
> def get_change(self, cnum=None, cdate=None, db=None):
> ...

Yes, I've seen some of that before. But looking that close and deep
didn't really provide the context I was looking for.


>> ...

>> When do other listeners get called for the change my listener saves?
>
> You'll see by now: They're executed right after altering the ticket data
> in the db. Straight from that springs the advice to not alter tickets in
> ticket change listeners, at least not in such a way, that a change will
> trigger another change and that another one and ...
>
> It'll be done ticket by ticket, but by re-spawning you could end up
> applying side-effects for a ticket that is actually waiting to get
> altered directly too.

Yeah. That's both clear and kind of icky for me. ;-)

If my listener is a singleton, I can test a global
"alreadyRescheduling" flag on entry, skip rescheduling if set, or set,
reschedule, and clear otherwise. I realize that may be a bit weak.

>...

> I've been reading your questions and explanation already for a while
> here. Now I'm forced to disclose some of my personal opinions.

I'm happy to have the feedback from someone who groks Trac a lot
better than I do.

> Take with a grain of salt, and listen to others as well.
>
> Doing automatic ticket changes for your PM stuff will not work well, if
> at all. Doing regular ticket changes each re-scheduling would leave a
> changes entry and change comment for each action to each ticket, and
> your tickets history will quickly grow beyond usable limits.

A valid point (and one already raised by a co-worker) but I'm still
experimenting. If that's true, a listener that does:

* recompute schedule
* update tickets

can easily morph to:

* recompute schedule
* update another table

> I conclude, that you'll likely need to
> * mess directly with PM-related ticket field values and spare the
> planning history
> * do it in a dedicated table outside of Trac db tables 'ticket' and
> 'ticket_custom' (preferred).

I think that the history is important and I'm not averse to an custom table.

> ...

Thanks for the feedback.

[Remy Blank]

Christopher Nelson wrote:
> But more
> broadly, I don't know if Trac gets executed in a single thread (so
> only one instance of my listener could be active at a time) or if,
> say, each user gets a thread in the server and multiple saves can
> fight.

Trac executes in several threads *and* several processes. It actually
depends on the configuration of your web server, but you should assume
the worst case.

-- Remy

[Christopher Nelson]

Of course. Back to the drawing board. :-)

[Christopher Nelson]

On Fri, Sep 14, 2012 at 3:09 PM, Christopher Nelson
<chris.ne...@gmail.com> wrote:
> On Fri, Sep 14, 2012 at 2:16 PM, Steffen Hoffmann <hof...@web.de> wrote:

> ...

>> Doing automatic ticket changes for your PM stuff will not work well, if
>> at all. Doing regular ticket changes each re-scheduling would leave a
>> changes entry and change comment for each action to each ticket, and
>> your tickets history will quickly grow beyond usable limits.
>
> A valid point (and one already raised by a co-worker) but I'm still
> experimenting. If that's true, a listener that does:
>
> * recompute schedule
> * update tickets
>
> can easily morph to:
>
> * recompute schedule
> * update another table

>...

So, I need to create my private tables. I got a pointer to
IEnvironmentSetupParticipant [1] and the code in Trac that creates the
database [2]. The former leads me to more detail on creating tables
[3]. I find a schema [4], but that module doesn't reference the
schema. Presumably there's some indirection I'm not following that
uses a DatabaseManager or something. What do I need to do in my
SetupParticipant to use schema to drive table creation?

Chris

[1] http://trac.edgewall.org/wiki/TracDev/PluginDevelopment/ExtensionPoints/trac.env.IEnvironmentSetupParticipant

[2] http://trac.edgewall.org/browser/trunk/trac/env.py#L556

[3] http://trac.edgewall.org/browser/trunk/trac/db_default.py

[4] http://trac.edgewall.org/browser/trunk/trac/db_default.py#L36

[Ethan Jucovy]

On Mon, Sep 17, 2012 at 10:28 AM, Christopher Nelson <chris.ne...@gmail.com> wrote:

So, I need to create my private tables.  I got a pointer to

IEnvironmentSetupParticipant [1] and the code in Trac that creates the
database [2].  The former leads me to more detail on creating tables
[3].  I find a schema [4], but that module doesn't reference the
schema.  Presumably there's some indirection I'm not following that
uses a DatabaseManager or something.  What do I need to do in my
SetupParticipant to use schema to drive table creation?

That schema[4] that you found is used during `trac-admin /path initenv`; once you trace through trac/admin/console.py the relevant code ends up being http://trac.edgewall.org/browser/trunk/trac/db/api.py#L247 and the db-backend-specific implementations in e.g. http://trac.edgewall.org/browser/trunk/trac/db/sqlite_backend.py#L208

For setting up database tables in plugins, I've used TracHoursPlugin as an example/template/thing-to-cargo-cult-from.  It uses a helper library (TracSqlHelperScript) that abstracts out a create_table function, but if you don't feel like making that a dependency of your plugin, it's only a few lines of code that you can copy over; your code will end up looking something like:

{{{

from trac.db import Table, Column, Index, DatabaseManager

class MySetupParticipant(Component):

    [...]

   

    def upgrade_environment(self, db):

        if i_should_not_create_tables(): return

        repo_version_table = Table('repository_version', key=('id'))[

            Column('id', auto_increment=True),

            Column('repo'),

            Column('version'),

            ]

        db_connector, _ = DatabaseManager(self.env)._get_connector()

        stmts = db_connector.to_sql(repo_version_table)

        cursor = db.cursor()

        for stmt in stmts:

            cursor.execute(stmt)

}}}

(Untested and probably includes some stupid typos.)

I'm not aware of any more formal core API for executing CREATE TABLE statements but would love to be wrong about that.

-Ethan

[Ethan Jucovy]

On Mon, Sep 17, 2012 at 11:12 AM, Ethan Jucovy <ethan....@gmail.com> wrote:

For setting up database tables in plugins, I've used TracHoursPlugin as an example/template/thing-to-cargo-cult-from.  It uses a helper library (TracSqlHelperScript) that abstracts out a create_table function, but if you don't feel like making that a dependency of your plugin, it's only a few lines of code that you can copy over; your code will end up looking something like [snip]

Or, if you *do* feel like just adding a dependency on TracSqlHelperScript, like I ended up doing, you could basically just copy all of the setup code in MultiRepoSearchPlugin here[1] and adjusting it for your needs.  In addition to demonstrating schema definition and table creation, it also uses Trac's system table to track what upgrades need to occur, and a neat little upgrade-steps-based-on-current-version framework thingie that I copied from TracHours.  (That's the `version()` and `steps =` stuff.)  I *think* this is considered The Right Way to manage upgrades in Trac plugins these days.

[1] https://github.com/boldprogressives/trac-MultiRepoSearchPlugin/blob/master/multireposearch/sqlindexer.py  

[Christopher Nelson]

> On Mon, Sep 17, 2012 at 11:12 AM, Ethan Jucovy <ethan....@gmail.com>
> wrote:

> ...

> Or, if you *do* feel like just adding a dependency on TracSqlHelperScript,
> like I ended up doing, you could basically just copy all of the setup code
> in MultiRepoSearchPlugin here[1] and adjusting it for your needs. In
> addition to demonstrating schema definition and table creation, it also uses
> Trac's system table to track what upgrades need to occur, and a neat little
> upgrade-steps-based-on-current-version framework thingie that I copied from
> TracHours. (That's the `version()` and `steps =` stuff.) I *think* this is
> considered The Right Way to manage upgrades in Trac plugins these days.

Thanks. I found an example in Subtickets plugin and will look at
TracHours, too.

[Christopher Nelson]

On Fri, Sep 14, 2012 at 3:09 PM, Christopher Nelson
<chris.ne...@gmail.com> wrote:

So, I have a working prototype but I'm struggling with the names of my
tables. Judging from MasterTickets and Subtickets plugins, there
doesn't seem to be common idiom or convention for naming tables to
avoid name conflicts with other plugins.

My tables will reside in the Trac database (environment) so "schedule"
seems a bad name, something someone else (or a future revision to
Trac) might want to use. I doubt there's an RDBMS-agnostic namespace
mechanism so I'm left with some plugin-specific prefix like
"pm_schedule" or "TracPM-schedule" or something. Can someone think of
a plugin that tries to be friendlier to other plugins so I can copy
its technique and start a trend?


Chris

[Christopher Nelson]

On Wed, Sep 19, 2012 at 9:38 PM, Christopher Nelson

<chris.ne...@gmail.com> wrote:
> On Fri, Sep 14, 2012 at 3:09 PM, Christopher Nelson
> <chris.ne...@gmail.com> wrote:

>...

>> I think that the history is important and I'm not averse to an custom
>> table.
>> ...
>
> So, I have a working prototype but I'm struggling with the names of my
> tables. Judging from MasterTickets and Subtickets plugins, there
> doesn't seem to be common idiom or convention for naming tables to
> avoid name conflicts with other plugins.
>
> My tables will reside in the Trac database (environment) so "schedule"
> seems a bad name, something someone else (or a future revision to
> Trac) might want to use. I doubt there's an RDBMS-agnostic namespace
> mechanism so I'm left with some plugin-specific prefix like
> "pm_schedule" or "TracPM-schedule" or something. Can someone think of
> a plugin that tries to be friendlier to other plugins so I can copy
> its technique and start a trend?

Still looking for table naming advice but I'm now keeping schedule
history in a second private table and I can reschedule 4k tickets in
25 seconds. In most cases, pruning to active tickets will make it
much, much faster so I think I'm good.

[Christopher Nelson]

Any advice on how to add fields to a private table in an upgrade script?

[Steffen Hoffmann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

By fields you mean columns, right?

Because ALTER TABLE doesn't work for all supported backends equally
well, in general you'll want to
* create a new temporary table as duplicating the existing one
* delete existing table
* re-create table with additional column
* populate new table from copy of previous table
* delete copy of previous table

That's the general pattern, that I've found in Trac core as well as in
some plugins. I've done it lately too [1], and with unit test coverage
for such upgrade modules I'm quite sure, that it works.

Steffen Hoffmann


[1]
http://trac-hacks.org/browser/announcerplugin/trunk/announcer/upgrades/db2.py?rev=12298

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAlCqpakACgkQ31DJeiZFuHcWMgCgisPwQyLgcw7rXda0Fj/Rr6o/
wgwAoLG+OZr9gqC6P9BnISLucIO0d08m
=AZq7
-----END PGP SIGNATURE-----

[Christopher Nelson]

>>>> ...
>>>> Or, if you *do* feel like just adding a dependency on TracSqlHelperScript,
>>>> like I ended up doing, you could basically just copy all of the setup code
>>>> in MultiRepoSearchPlugin here[1] and adjusting it for your needs. In
>>>> addition to demonstrating schema definition and table creation, it also uses
>>>> Trac's system table to track what upgrades need to occur, and a neat little
>>>> upgrade-steps-based-on-current-version framework thingie that I copied from
>>>> TracHours. (That's the `version()` and `steps =` stuff.) I *think* this is
>>>> considered The Right Way to manage upgrades in Trac plugins these days.
>>>
>>> Thanks. I found an example in Subtickets plugin and will look at
>>> TracHours, too.
>>
>> Any advice on how to add fields to a private table in an upgrade script?
>
> By fields you mean columns, right?

Yes.

> Because ALTER TABLE doesn't work for all supported backends equally
> well, in general you'll want to
> * create a new temporary table as duplicating the existing one
> * delete existing table
> * re-create table with additional column
> * populate new table from copy of previous table
> * delete copy of previous table
>
> That's the general pattern, that I've found in Trac core as well as in
> some plugins. I've done it lately too [1], and with unit test coverage
> for such upgrade modules I'm quite sure, that it works.
>
> Steffen Hoffmann

Thanks. What I found in SubticketsPlugin was:

* Copy the data from the existing tables into Python hashes
* Drop the tables
* Create the tables for the new schema
* Insert the saved data

Interestingly, there was no logic to only do this if the schema was
out of date. Maybe that was OK there but it seemed wrong. In my
adaptation, I only save and create if the version changed. Seems to
work. So far. <fingers crossed>

[Steffen Hoffmann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 19.11.2012 22:36, Christopher Nelson wrote:
> What I found in SubticketsPlugin was:
>
> * Copy the data from the existing tables into Python hashes
> * Drop the tables
> * Create the tables for the new schema
> * Insert the saved data
>
> Interestingly, there was no logic to only do this if the schema was
> out of date. Maybe that was OK there but it seemed wrong.

Rebuild table on each check == environment load, really? That wouldn't
be wrong, but totally insane. Happen, that I use this plugin myself,
I'll have to check this for sure.

> In my adaptation, I only save and create if the version changed. Seems to
> work. So far. <fingers crossed>

Sure, nothing more is required.

Steffen Hoffmann

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAlCqz0oACgkQ31DJeiZFuHesZACfUWMagWu4KH8Lv36lycTG8uQB
fFcAoIU+Et26916Y0zcuhNY8gyGicq7v
=knTg
-----END PGP SIGNATURE-----