Developer mailing list for the Trac Project

Christopher Nelson | 14 Sep 2012 15:19

[Trac-dev] ITicketChangeListener semantics

http://www.edgewall.org/docs/branches-0.12-stable/epydoc/trac.ticket.api.ITicketChangeListener-class.html
shows me the syntax/interface for a ticket change listener but doesn't
give me a lot of context for when it is called, what I can safely do
in a listener, etc.  I'm running into a problem in my listener that
only shows up on large datasets in our test environment but not in the
small dataset in my development environment and I'd like to understand
timing and context so I know where to look for my problem.  Is there
some tutorial or other documentation that will tell me how listeners
interact with the system and with each other?  For example, if my
listener saves a ticket, does it get called again recursively so I
have to be reentrant?  Does another call get deferred until this call
ends?  When do other listeners get called for the change my listener
saves?

Here's what I'm trying to do: when a field related to scheduling
tickets changes, do a minimal schedule recalculation.  So, if the
estimate for a ticket change from 8 hours to 16, any following work
will start a day later or this task will start a day earlier.  In the
latter case, I'll do all my calculations then have a new start date
for the ticket that the listener got invoked for.  When I save that
change -- inside the listener -- what happens?

                                                                        Chris
-- 
A: Top-posting.
Q: What is the most annoying thing in e-mail?

--

-- 
You received this message because you are subscribed to the Google Groups "Trac Development" group.
To post to this group, send email to trac-dev <at> googlegroups.com.

(Continue reading)

headers

Peter Suter | 14 Sep 2012 17:32

Re: [Trac-dev] ITicketChangeListener semantics

On 14.09.2012 15:19, Christopher Nelson wrote:
> Is there
> some tutorial or other documentation that will tell me how listeners
> interact with the system and with each other?

One place where I've been trying to document such things is in [1]. 
Unfortunately the ITicketChangeListener page is not up yet.

I'm not aware of any other such resource. Unless the Trac source code 
counts. :)

>  For example, if my
> listener saves a ticket, does it get called again recursively so I
> have to be reentrant?  Does another call get deferred until this call
> ends?  When do other listeners get called for the change my listener
> saves?
 >
 > Here's what I'm trying to do: when a field related to scheduling
 > tickets changes, do a minimal schedule recalculation.  So, if the
 > estimate for a ticket change from 8 hours to 16, any following work
 > will start a day later or this task will start a day earlier.  In the
 > latter case, I'll do all my calculations then have a new start date
 > for the ticket that the listener got invoked for.  When I save that
 > change -- inside the listener -- what happens?

All ITicketChangeListener.ticket_changed() implementations get called in 
a loop at the end of Ticket.save_changes() [2].

There is no special mechanism to handle reentrancy. 
ITicketChangeListener is not meant to be used in situations where the

(Continue reading)

headers

Christopher Nelson | 14 Sep 2012 21:04

Re: [Trac-dev] ITicketChangeListener semantics

> On 14.09.2012 15:19, Christopher Nelson wrote:
>> Is there
>> some tutorial or other documentation that will tell me how listeners
>> interact with the system and with each other?
>
> One place where I've been trying to document such things is in [1].
> Unfortunately the ITicketChangeListener page is not up yet.

I appreciate you efforts and have been there a few times.  Thanks.

> I'm not aware of any other such resource. Unless the Trac source code
> counts. :)

Of course it does.  And I had seen where it gets invoked.  But more
broadly, I don't know if Trac gets executed in a single thread (so
only one instance of my listener could be active at a time) or if,
say, each user gets a thread in the server and multiple saves can
fight.

>...
>> Here's what I'm trying to do: when a field related to scheduling
>> tickets changes, do a minimal schedule recalculation.  So, if the
>> estimate for a ticket change from 8 hours to 16, any following work
>> will start a day later or this task will start a day earlier.  In the
>> latter case, I'll do all my calculations then have a new start date
>> for the ticket that the listener got invoked for.  When I save that
>> change -- inside the listener -- what happens?
>
> All ITicketChangeListener.ticket_changed() implementations get called in a
> loop at the end of Ticket.save_changes() [2].

(Continue reading)

headers

Remy Blank | 14 Sep 2012 22:12

Re: [Trac-dev] ITicketChangeListener semantics

Christopher Nelson wrote:
> But more
> broadly, I don't know if Trac gets executed in a single thread (so
> only one instance of my listener could be active at a time) or if,
> say, each user gets a thread in the server and multiple saves can
> fight.

Trac executes in several threads *and* several processes. It actually
depends on the configuration of your web server, but you should assume
the worst case.

-- Remy

headers

Christopher Nelson | 14 Sep 2012 22:19

Re: [Trac-dev] ITicketChangeListener semantics

> Christopher Nelson wrote:
>> But more
>> broadly, I don't know if Trac gets executed in a single thread (so
>> only one instance of my listener could be active at a time) or if,
>> say, each user gets a thread in the server and multiple saves can
>> fight.
>
> Trac executes in several threads *and* several processes. It actually
> depends on the configuration of your web server, but you should assume
> the worst case.

Of course.  Back to the drawing board. 

--

-- 
You received this message because you are subscribed to the Google Groups "Trac Development" group.
To post to this group, send email to trac-dev <at> googlegroups.com.
To unsubscribe from this group, send email to trac-dev+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/trac-dev?hl=en.

headers

Steffen Hoffmann | 14 Sep 2012 20:16

Re: [Trac-dev] ITicketChangeListener semantics

On 14.09.2012 15:19, Christopher Nelson wrote:
> http://www.edgewall.org/docs/branches-0.12-stable/epydoc/trac.ticket.api.ITicketChangeListener-class.html
>
> 
shows me the syntax/interface for a ticket change listener but doesn't
> give me a lot of context for when it is called,

A full-text search over Trac sources yields only one place,
trac.ticket.api, where ITicketChangeListener is mentioned (plus one
occurrence in trac.ticket.tests.model - a unit test).

But note:
 trac.ticket.api.TicketSystem.

A closer look at trac.ticket.model.Ticket reveals the core logic,that is
executed in appropriate methods there like so:

class Ticket(object):
    ...

    def delete(self, db=None):
        ...

        for listener in TicketSystem(self.env).change_listeners:
            listener.ticket_deleted(self)

    def get_change(self, cnum=None, cdate=None, db=None):
        ...

(Continue reading)

headers

Steffen Hoffmann | 14 Sep 2012 20:19

Re: [Trac-dev] ITicketChangeListener semantics


On 14.09.2012 20:16, Steffen Hoffmann wrote:
> But note:
>  trac.ticket.api.TicketSystem.

Oh, this was unfinished. I meant to point at this:

    change_listeners = ExtensionPoint(ITicketChangeListener)
(in trac.ticket.api.TicketSystem)

Steffen Hoffmann

headers

Christopher Nelson | 14 Sep 2012 21:09

Re: [Trac-dev] ITicketChangeListener semantics

On Fri, Sep 14, 2012 at 2:16 PM, Steffen Hoffmann <hoff.st <at> web.de> wrote:
> On 14.09.2012 15:19, Christopher Nelson wrote:
> A full-text search over Trac sources yields only one place,
> trac.ticket.api, where ITicketChangeListener is mentioned (plus one
> occurrence in trac.ticket.tests.model - a unit test).
>
> But note:
>  trac.ticket.api.TicketSystem.
>
> A closer look at trac.ticket.model.Ticket reveals the core logic,that is
> executed in appropriate methods there like so:
>
> class Ticket(object):
>     ...
>
>     def delete(self, db=None):
>         ...
>
>         for listener in TicketSystem(self.env).change_listeners:
>             listener.ticket_deleted(self)
>
>     def get_change(self, cnum=None, cdate=None, db=None):
>         ...

Yes, I've seen some of that before.  But looking that close and deep
didn't really provide the context I was looking for.

>> ...
>> When do other listeners get called for the change my listener saves?
>

(Continue reading)

headers

Christopher Nelson | 17 Sep 2012 16:28

Re: [Trac-dev] ITicketChangeListener semantics

On Fri, Sep 14, 2012 at 3:09 PM, Christopher Nelson
<chris.nelson.1022 <at> gmail.com> wrote:
> On Fri, Sep 14, 2012 at 2:16 PM, Steffen Hoffmann <hoff.st <at> web.de> wrote:
> ...
>> Doing automatic ticket changes for your PM stuff will not work well, if
>> at all. Doing regular ticket changes each re-scheduling would leave a
>> changes entry and change comment for each action to each ticket, and
>> your tickets history will quickly grow beyond usable limits.
>
> A valid point (and one already raised by a co-worker) but I'm still
> experimenting.  If that's true, a listener that does:
>
>  * recompute schedule
>  * update tickets
>
> can easily morph to:
>
>  * recompute schedule
>  * update another table
>...

So, I need to create my private tables.  I got a pointer to
IEnvironmentSetupParticipant [1] and the code in Trac that creates the
database [2].  The former leads me to more detail on creating tables
[3].  I find a schema [4], but that module doesn't reference the
schema.  Presumably there's some indirection I'm not following that
uses a DatabaseManager or something.  What do I need to do in my
SetupParticipant to use schema to drive table creation?

                                                                         Chris

(Continue reading)

headers

Ethan Jucovy | 17 Sep 2012 17:12

Re: [Trac-dev] ITicketChangeListener semantics

On Mon, Sep 17, 2012 at 10:28 AM, Christopher Nelson <chris.nelson.1022 <at> gmail.com> wrote:

So, I need to create my private tables. I got a pointer to
IEnvironmentSetupParticipant [1] and the code in Trac that creates the
database [2]. The former leads me to more detail on creating tables
[3]. I find a schema [4], but that module doesn't reference the
schema. Presumably there's some indirection I'm not following that
uses a DatabaseManager or something. What do I need to do in my
SetupParticipant to use schema to drive table creation?

That schema[4] that you found is used during `trac-admin /path initenv`; once you trace through trac/admin/console.py the relevant code ends up being http://trac.edgewall.org/browser/trunk/trac/db/api.py#L247 and the db-backend-specific implementations in e.g. http://trac.edgewall.org/browser/trunk/trac/db/sqlite_backend.py#L208

For setting up database tables in plugins, I've used TracHoursPlugin as an example/template/thing-to-cargo-cult-from. It uses a helper library (TracSqlHelperScript) that abstracts out a create_table function, but if you don't feel like making that a dependency of your plugin, it's only a few lines of code that you can copy over; your code will end up looking something like:

{{{

from trac.db import Table, Column, Index, DatabaseManager

class MySetupParticipant(Component):

[...]

def upgrade_environment(self, db):

if i_should_not_create_tables(): return

repo_version_table = Table('repository_version', key=('id'))[

Column('id', auto_increment=True),

Column('repo'),

Column('version'),

]

db_connector, _ = DatabaseManager(self.env)._get_connector()

stmts = db_connector.to_sql(repo_version_table)

cursor = db.cursor()

for stmt in stmts:

cursor.execute(stmt)

}}}

(Untested and probably includes some stupid typos.)

I'm not aware of any more formal core API for executing CREATE TABLE statements but would love to be wrong about that.

-Ethan

[1] http://trac.edgewall.org/wiki/TracDev/PluginDevelopment/ExtensionPoints/trac.env.IEnvironmentSetupParticipant

[2] http://trac.edgewall.org/browser/trunk/trac/env.py#L556

[3] http://trac.edgewall.org/browser/trunk/trac/db_default.py

[4] http://trac.edgewall.org/browser/trunk/trac/db_default.py#L36

--
You received this message because you are subscribed to the Google Groups "Trac Development" group.
To post to this group, send email to trac-dev <at> googlegroups.com.
To unsubscribe from this group, send email to trac-dev+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/trac-dev?hl=en.

headers

Ethan Jucovy | 17 Sep 2012 17:19

Re: [Trac-dev] ITicketChangeListener semantics

On Mon, Sep 17, 2012 at 11:12 AM, Ethan Jucovy <ethan.jucovy <at> gmail.com> wrote:

For setting up database tables in plugins, I've used TracHoursPlugin as an example/template/thing-to-cargo-cult-from. It uses a helper library (TracSqlHelperScript) that abstracts out a create_table function, but if you don't feel like making that a dependency of your plugin, it's only a few lines of code that you can copy over; your code will end up looking something like [snip]

Or, if you *do* feel like just adding a dependency on TracSqlHelperScript, like I ended up doing, you could basically just copy all of the setup code in MultiRepoSearchPlugin here[1] and adjusting it for your needs. In addition to demonstrating schema definition and table creation, it also uses Trac's system table to track what upgrades need to occur, and a neat little upgrade-steps-based-on-current-version framework thingie that I copied from TracHours. (That's the `version()` and `steps =` stuff.) I *think* this is considered The Right Way to manage upgrades in Trac plugins these days.

[1] https://github.com/boldprogressives/trac-MultiRepoSearchPlugin/blob/master/multireposearch/sqlindexer.py

headers

Christopher Nelson | 17 Sep 2012 17:25

Re: [Trac-dev] ITicketChangeListener semantics

> On Mon, Sep 17, 2012 at 11:12 AM, Ethan Jucovy <ethan.jucovy <at> gmail.com>
> wrote:
> ...
> Or, if you *do* feel like just adding a dependency on TracSqlHelperScript,
> like I ended up doing, you could basically just copy all of the setup code
> in MultiRepoSearchPlugin here[1] and adjusting it for your needs.  In
> addition to demonstrating schema definition and table creation, it also uses
> Trac's system table to track what upgrades need to occur, and a neat little
> upgrade-steps-based-on-current-version framework thingie that I copied from
> TracHours.  (That's the `version()` and `steps =` stuff.)  I *think* this is
> considered The Right Way to manage upgrades in Trac plugins these days.

Thanks.  I found an example in Subtickets plugin and will look at
TracHours, too.

--

-- 
You received this message because you are subscribed to the Google Groups "Trac Development" group.
To post to this group, send email to trac-dev <at> googlegroups.com.
To unsubscribe from this group, send email to trac-dev+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/trac-dev?hl=en.

headers

Christopher Nelson | 19 Nov 2012 20:26

Re: [Trac-dev] ITicketChangeListener semantics

>> On Mon, Sep 17, 2012 at 11:12 AM, Ethan Jucovy <ethan.jucovy <at> gmail.com>
>> wrote:
>> ...
>> Or, if you *do* feel like just adding a dependency on TracSqlHelperScript,
>> like I ended up doing, you could basically just copy all of the setup code
>> in MultiRepoSearchPlugin here[1] and adjusting it for your needs.  In
>> addition to demonstrating schema definition and table creation, it also uses
>> Trac's system table to track what upgrades need to occur, and a neat little
>> upgrade-steps-based-on-current-version framework thingie that I copied from
>> TracHours.  (That's the `version()` and `steps =` stuff.)  I *think* this is
>> considered The Right Way to manage upgrades in Trac plugins these days.
>
> Thanks.  I found an example in Subtickets plugin and will look at
> TracHours, too.

Any advice on how to add fields to a private table in an upgrade script?

-- 
A: Top-posting.
Q: What is the most annoying thing in e-mail?

--

-- 
You received this message because you are subscribed to the Google Groups "Trac Development" group.
To post to this group, send email to trac-dev <at> googlegroups.com.
To unsubscribe from this group, send email to trac-dev+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/trac-dev?hl=en.

headers

Steffen Hoffmann | 19 Nov 2012 22:33

Re: [Trac-dev] ITicketChangeListener semantics


On 19.11.2012 20:26, Christopher Nelson wrote:
>>> On Mon, Sep 17, 2012 at 11:12 AM, Ethan Jucovy <ethan.jucovy <at> gmail.com>
>>> wrote:
>>> ...
>>> Or, if you *do* feel like just adding a dependency on TracSqlHelperScript,
>>> like I ended up doing, you could basically just copy all of the setup code
>>> in MultiRepoSearchPlugin here[1] and adjusting it for your needs.  In
>>> addition to demonstrating schema definition and table creation, it also uses
>>> Trac's system table to track what upgrades need to occur, and a neat little
>>> upgrade-steps-based-on-current-version framework thingie that I copied from
>>> TracHours.  (That's the `version()` and `steps =` stuff.)  I *think* this is
>>> considered The Right Way to manage upgrades in Trac plugins these days.
>>
>> Thanks.  I found an example in Subtickets plugin and will look at
>> TracHours, too.
> 
> Any advice on how to add fields to a private table in an upgrade script?

By fields you mean columns, right?

Because ALTER TABLE doesn't work for all supported backends equally
well, in general you'll want to
 * create a new temporary table as duplicating the existing one
 * delete existing table
 * re-create table with additional column
 * populate new table from copy of previous table
 * delete copy of previous table

That's the general pattern, that I've found in Trac core as well as in
some plugins. I've done it lately too [1], and with unit test coverage
for such upgrade modules I'm quite sure, that it works.

Steffen Hoffmann

[1]
http://trac-hacks.org/browser/announcerplugin/trunk/announcer/upgrades/db2.py?rev=12298

headers

Christopher Nelson | 19 Nov 2012 22:36

Re: [Trac-dev] ITicketChangeListener semantics

>>>> ...
>>>> Or, if you *do* feel like just adding a dependency on TracSqlHelperScript,
>>>> like I ended up doing, you could basically just copy all of the setup code
>>>> in MultiRepoSearchPlugin here[1] and adjusting it for your needs.  In
>>>> addition to demonstrating schema definition and table creation, it also uses
>>>> Trac's system table to track what upgrades need to occur, and a neat little
>>>> upgrade-steps-based-on-current-version framework thingie that I copied from
>>>> TracHours.  (That's the `version()` and `steps =` stuff.)  I *think* this is
>>>> considered The Right Way to manage upgrades in Trac plugins these days.
>>>
>>> Thanks.  I found an example in Subtickets plugin and will look at
>>> TracHours, too.
>>
>> Any advice on how to add fields to a private table in an upgrade script?
>
> By fields you mean columns, right?

Yes.

> Because ALTER TABLE doesn't work for all supported backends equally
> well, in general you'll want to
>  * create a new temporary table as duplicating the existing one
>  * delete existing table
>  * re-create table with additional column
>  * populate new table from copy of previous table
>  * delete copy of previous table
>
> That's the general pattern, that I've found in Trac core as well as in
> some plugins. I've done it lately too [1], and with unit test coverage
> for such upgrade modules I'm quite sure, that it works.
>
> Steffen Hoffmann

Thanks.  What I found in SubticketsPlugin was:

 * Copy the data from the existing tables into Python hashes
 * Drop the tables
 * Create the tables for the new schema
 * Insert the saved data

Interestingly, there was no logic to only do this if the schema was
out of date.  Maybe that was OK there but it seemed wrong.  In my
adaptation, I only save and create if the version changed.  Seems to
work.  So far. <fingers crossed>

--

-- 
You received this message because you are subscribed to the Google Groups "Trac Development" group.
To post to this group, send email to trac-dev <at> googlegroups.com.
To unsubscribe from this group, send email to trac-dev+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/trac-dev?hl=en.

headers

Steffen Hoffmann | 20 Nov 2012 01:31

Re: [Trac-dev] ITicketChangeListener semantics


On 19.11.2012 22:36, Christopher Nelson wrote:
> What I found in SubticketsPlugin was:
> 
>  * Copy the data from the existing tables into Python hashes
>  * Drop the tables
>  * Create the tables for the new schema
>  * Insert the saved data
> 
> Interestingly, there was no logic to only do this if the schema was
> out of date.  Maybe that was OK there but it seemed wrong.

Rebuild table on each check == environment load, really? That wouldn't
be wrong, but totally insane. Happen, that I use this plugin myself,
I'll have to check this for sure.

> In my adaptation, I only save and create if the version changed.  Seems to
> work.  So far. <fingers crossed>

Sure, nothing more is required.

Steffen Hoffmann

headers

Christopher Nelson | 20 Sep 2012 03:38

Re: [Trac-dev] ITicketChangeListener semantics

On Fri, Sep 14, 2012 at 3:09 PM, Christopher Nelson
<chris.nelson.1022 <at> gmail.com> wrote:
> On Fri, Sep 14, 2012 at 2:16 PM, Steffen Hoffmann <hoff.st <at> web.de> wrote:
> > On 14.09.2012 15:19, Christopher Nelson wrote:
> > Doing automatic ticket changes for your PM stuff will not work well, if
> > at all. Doing regular ticket changes each re-scheduling would leave a
> > changes entry and change comment for each action to each ticket, and
> > your tickets history will quickly grow beyond usable limits.
>
> A valid point (and one already raised by a co-worker) but I'm still
> experimenting.  If that's true, a listener that does:
>
>  * recompute schedule
>  * update tickets
>
> can easily morph to:
>
>  * recompute schedule
>  * update another table
>
> > I conclude, that you'll likely need to
> >  * mess directly with PM-related ticket field values and spare the
> > planning history
> >  * do it in a dedicated table outside of Trac db tables 'ticket' and
> > 'ticket_custom' (preferred).
>
> I think that the history is important and I'm not averse to an custom
> table.
> ...

So, I have a working prototype but I'm struggling with the names of my
tables.  Judging from MasterTickets and Subtickets plugins, there
doesn't seem to be common idiom or convention for naming tables to
avoid name conflicts with other plugins.

My tables will reside in the Trac database (environment) so "schedule"
seems a bad name, something someone else (or a future revision to
Trac) might want to use.  I doubt there's an RDBMS-agnostic namespace
mechanism so I'm left with some plugin-specific prefix like
"pm_schedule" or "TracPM-schedule" or something.  Can someone think of
a plugin that tries to be friendlier to other plugins so I can copy
its technique and start a trend?

     Chris

--

-- 
You received this message because you are subscribed to the Google Groups "Trac Development" group.
To post to this group, send email to trac-dev <at> googlegroups.com.
To unsubscribe from this group, send email to trac-dev+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/trac-dev?hl=en.

headers

Christopher Nelson | 21 Sep 2012 16:20

Re: [Trac-dev] ITicketChangeListener semantics

On Wed, Sep 19, 2012 at 9:38 PM, Christopher Nelson
<chris.nelson.1022 <at> gmail.com> wrote:
> On Fri, Sep 14, 2012 at 3:09 PM, Christopher Nelson
> <chris.nelson.1022 <at> gmail.com> wrote:
>...
>> I think that the history is important and I'm not averse to an custom
>> table.
>> ...
>
> So, I have a working prototype but I'm struggling with the names of my
> tables.  Judging from MasterTickets and Subtickets plugins, there
> doesn't seem to be common idiom or convention for naming tables to
> avoid name conflicts with other plugins.
>
> My tables will reside in the Trac database (environment) so "schedule"
> seems a bad name, something someone else (or a future revision to
> Trac) might want to use.  I doubt there's an RDBMS-agnostic namespace
> mechanism so I'm left with some plugin-specific prefix like
> "pm_schedule" or "TracPM-schedule" or something.  Can someone think of
> a plugin that tries to be friendlier to other plugins so I can copy
> its technique and start a trend?

Still looking for table naming advice but I'm now keeping schedule
history in a second private table and I can reschedule 4k tickets in
25 seconds.  In most cases, pruning to active tickets will make it
much, much faster so I think I'm good.

--

-- 
You received this message because you are subscribed to the Google Groups "Trac Development" group.
To post to this group, send email to trac-dev <at> googlegroups.com.
To unsubscribe from this group, send email to trac-dev+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/trac-dev?hl=en.