Trac internals' database simplification

[Javier Domingo]

Hi, I am looking into trac's code, and I have found that there is no DAL. It has a very flexible database design, but it is not an efficient design for a database that will have lots of queries.

I have found that it uses hard-coded queries (which is difficult to understand how it works), text as primary keys, etc.

I would like to ask why doesn't trac use django (with south for db versioning).

I plan to develop a parallel ticket system as a plugin (as I mentioned in previous threads) using as base the original trac.ticket structure, using django as a DAL and maintaining the interfaces.

I say all this because I think that trac is a very great software if you want to extend it, the interface is clear and simple, but I think it needs to be internally hackable.

Looking back for your replies,

Javier Domingo

[Leho Kraav]

Hi Javier

Trac is almost 10 years old. Django (or anything else, Pyramid etc) wasn't nearly the quality players they are today. Trac is also a web application framework on its own. It has therefore never needed an outside framework to perform its job.

Main implementation on top of this app is a rather specialized software development process tool. Because the amount of core developers has remained small for whatever reason, no (human) resources have appeared that could've made Trac web app framework core part to the level of Django so whatever apps could easily be built on top of Trac core. Very few people have needed this and it's a very difficult thing to do.

That being said, I myself long been interested in somehow either getting Trac core to a higher level (Wordpress, Django) so it could support building a wider range of applications sitting next to its excellenet main functionality

OR

Take Trac's primary modules (Wiki, Ticket) and port them on top of a generalized web app framework that has a significantly wider user and development base, so you don't have to write alot of plumbing again if you want to make a new app.

Unfortunately both of these tasks is a *significant* undertaking. It is not a trivial task at all. I'd estimate months of pretty serious time commitment for at least 2 skilled developers to achieve anything significant either way.

So, out of curiosity (for now), how serious are you about attempting this? Hacking interest can fade very quickly when tasks are hard.

[Hans-Peter Locher]

Hi,

+ 1 for creating/adding a 

DAL to Trac

I could imagine that doing a fundraising for the goal could raise some money

for solving the human ressource problem.

best regards

Hans-Peter Locher

--
You received this message because you are subscribed to the Google Groups "Trac Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trac-dev+u...@googlegroups.com.
To post to this group, send email to trac...@googlegroups.com.
Visit this group at http://groups.google.com/group/trac-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Leho Kraav]

+ 1 for creating/adding a 

DAL to Trac

I could imagine that doing a fundraising for the goal could raise some money

for solving the human ressource problem.

Maybe. Noone has attempted that yet. Anyone feeling like a Kickstarter project? :) I know I would support this for sure, and I'm not talking $5.

But Hans, which direction are you preferring. Porting Trac main modules -> let's say Django, or hack Trac's core framework up towards the giants.

My spidey sense is telling me world doesn't really need yet another web framework. I think it would make (at least for business) sense to port the modules on top of bigger frameworks.

[Javier Domingo]

I will try to explain a little which is my interest on this.

First of all, I have a project I have semi-launched (I am an student,
and my University is the client) a project, as part of a bigger one to
have a better communication between students and student council. As I
needed a very privacy-save ticketing system, I decided to clone the
actual one, and modify it.

The second project I am trying to launch for a pair of hospitals as a
innovation project, is making a set of apps that help managing
urgencies, using trac as backend.

For both cases, I will have to modify trac's ticketing system, and I
hope I get funds. Any way I will tell you when I get into work
seriously (I am still designing).

I am not an expert python developer, but I usually work out my ideas.

I am still trying to imagine how would it be to maintain the plugins
interface with a django framework. I have thought of making it a
django app, but I dropped that idea because I like the way you have to
manage plugins and functionalities.

I am thinking more on something like this:
http://stackoverflow.com/questions/579511/using-only-the-db-part-of-django

Javier Domingo

[Hans-Peter Locher]

-1 for hacking ;)

I wouldn't suggest re inventing the wheel.

I might prefer adding a widely used python orm like one of the following:

sqlalchemy

https://pypi.python.org/pypi/SQLAlchemy

storm

https://pypi.python.org/pypi/storm

peewee

https://pypi.python.org/pypi/peewee/1.0.0

...

over making trac a django based application.

As moving trac to become a django based app would have a large effect on

- addon compatibilty

- migration of exisitng trac instances

- addon development in general

Though it might enable django developers to develop trac addons more easily.

Me / the company I work for would also support a fundraising

as we use trac for our development process as well as platform for applications.

This subject needs a broad discussion in the trac community in my opinion.

Best regards

Hans-Peter Locher

[Leho Kraav]

On Tuesday, March 5, 2013 12:35:33 PM UTC+2, Javier Domingo wrote:

I am not an expert python developer, but I usually work out my ideas.

I am still trying to imagine how would it be to maintain the plugins
interface with a django framework. I have thought of making it a
django app, but I dropped that idea because I like the way you have to
manage plugins and functionalities.

I am thinking more on something like this:
http://stackoverflow.com/questions/579511/using-only-the-db-part-of-django

Mmmmkay. Well the thing is, for general purpose apps, Trac also has other issues besides just not having a high-level DAL. Even if you get it working on top of just another db layer, that's all good, but it will still leave a laaaaarge amount of manual work to be done for any general app feature.

There are issues with the existing schema for example (search GenericTrac) that should be improved as well. IMHO a django app (or Pyramid app? I don't know its structure, but I do know players are using it) would be a much more thorough and future proof solution than just trying to rotate a few foundation components underneath the highly specialized app implementation that Trac is today.

[Leho Kraav]

On Tuesday, March 5, 2013 12:45:36 PM UTC+2, Hans-Peter Locher wrote:

over making trac a django based application.

As moving trac to become a django based app would have a large effect on

- addon compatibilty

- migration of exisitng trac instances

- addon development in general

Absolutely.

But when you look at current trac-hacks catalog, there's only a few really serious must-haves there. I think they can be ported with reasonable effort, perhaps even simply replaced with existing Django ecosystem apps. All the other minor hacks it doesn't really make sense to worry about.

If it'd somehow be possible to achieve a new platform codebase that could just rebase and merge Trac's core modules (Tickets, Wiki) improvements, that'd be awesome.

There's also Apache BloodHound. I should dig through their mailing lists because I'm fairly certain they also discussed these options.

[Javier Domingo]

So, IMHO following Leho's opinion, the best implementation would be
converting trac in a django app, that allows the plugin-ing into it,
without modifying it.

In that way, a client would just create a new django project, activate
the trac app, and download/install the plugins as other django apps
that would interact with the original one using the ExtensionPoints.

Now, the only problem it comes to my mind is: How to manage
permissions? If you realize, a lot of people (if not nearly everyone)
is using the AccountManagerPlugin. So going Django would mean using
its permission system, along with their authentication system, which
can be totally customized (I love the way trac handles authn)

I think that going django would mean a complete remake of trac, but
would simplify a lot trac deployment (in for example appengine), and
would allow users to integrate it completely with their python web
pages.

[Brettschneider Falk]

Remember, once there was http://trac.edgewall.org/ticket/6339.

CU, F@lk

----
R&D Software
Baumer Optronic GmbH
www.baumer.com





Gesch?ftsf?hrer: Dr. Albert Schmidt* Dr. Oliver Vietze
Sitz der Gesellschaft: Radeberg
Amtsgericht Dresden: HRB 15379
Ust. ID: DE 189714583

[Leho Kraav]

On Tuesday, March 5, 2013 2:10:19 PM UTC+2, Brettschneider Falk wrote:

Remember, once there was http://trac.edgewall.org/ticket/6339.

Yep. It's pretty obvious there wasn't much demand there and the wontfix reason is probably valid. It's too much work for dubious benefit just to do it for the database alone.

[Leho Kraav]

That's a pretty good wrap-up. Yes, everybody's favorite, the good old user model question comes up now :) #2456

https://docs.djangoproject.com/en/1.5/releases/1.5/#configurable-user-model is something that caught my eye in last month's Django's release notes.

 
I so far think it's a feasible idea to replicate the permission system, so it would appear the same. Perhaps some customizable ACL system already exists for Django. AFAICT, whatever the implementation, on surface all reasonable permission systems end up looking pretty similar i.e. some kind of a list of named capabilities.

[Leho Kraav]

On Tuesday, March 5, 2013 2:00:14 PM UTC+2, Javier Domingo wrote:

So, IMHO following Leho's opinion, the best implementation would be
converting trac in a django app, that allows the plugin-ing into it,
without modifying it.

Well I'm far from sure it's the "best" option :) but I would be willing to invest a reasonable
amount of exploratory coding work time into trying to get a Trac core module working on Django.
Start going step by step, learning more about the issues as we go. If it's doomed to failure, it
should at least fail fast (general rule of any startup) and I wouldn't have a regret about it, perhaps
we shall discover a Better Way.

[Ethan Jucovy]

On Tue, Mar 5, 2013 at 7:00 AM, Javier Domingo <javi...@gmail.com> wrote:

So, IMHO following Leho's opinion, the best implementation would be
converting trac in a django app, that allows the plugin-ing into it,
without modifying it.

I agree with Hans-Peter that building a SQLAlchemy data layer (and GenericTrac, and a real user model) would be very interesting, if somebody is able to put in the effort. 

I'm not sure what would be gained by reimplementing Trac on top of a framework like Django.  If you want to build general purpose web applications you can already do it in Django.  IIRC Trac's core component architecture works fine on its own so you could even use Trac as a library for structuring your code in another web framework.  And it's not difficult to define Django models to interact with a Trac database (at least for reading data) if you want to build a custom-purpose web application to integrate with your Trac environment and just use Trac as a backend.  If the goal is to have a trac-like system that's a "pluggable app" you can drop into a larger system with deep integration, I don't think that rebuilding Trac as a Django application is likely to be any more drop-in-friendly than Trac already is -- write an authentication source and a theme, mount it as a wsgi application on some subpath, and then work out the kinks one by one.  :-)  After all, even most Django apps are not actually very drop-in friendly if they provide sufficiently high level features.

As far as the specific use case you've described, have you looked into Trac's permission policy system?  On your previous thread Peter Stuge suggested a possible route for ensuring security -- doing what Peter outlined, plus perhaps removing Trac's default permission policy altogether in your environments, should provide you with a secure-by-default system without having to reinvent everything.
 
-Ethan

[Peter Stuge]

Javier Domingo wrote:
> I think that going django would mean a complete remake of trac, but
> would simplify a lot trac deployment (in for example appengine)

Stop the madness.

Trac has existed for 10+ years. You're a student who doesn't really
know Python (you said).

It makes no sense whatsoever for you to rewrite Trac using hipster
frameworks!

You would be completely alone working on that, it would take an
enormous amount of time because you have to learn so many things,
and in the end it's not likely going to be any better than what is
already there.

Please just learn to develop the existing Trac instead, and work
together with existing community. You will be able to accomplish
much more with much less effort.

I'll echo Ethan's question: Have you already looked into my
suggestion to extend the existing permission system?

I think that would be a pretty simple solution to your problem (I'd
estimate one week if you can read and write Python) and I imagine
that your code would inspire other cool new ideas for how Trac
permissions can be used.


//Peter

[Javier Domingo]

2013/3/5 Peter Stuge <pe...@stuge.se>

Javier Domingo wrote:
> I think that going django would mean a complete remake of trac, but
> would simplify a lot trac deployment (in for example appengine)

Stop the madness.

Trac has existed for 10+ years. You're a student who doesn't really
know Python (you said).

I don't know which is the level you can say that really know a language.​​

It makes no sense whatsoever for you to rewrite Trac using hipster
frameworks!

You would be completely alone working on that, it would take an
enormous amount of time because you have to learn so many things,
and in the end it's not likely going to be any better than what is
already there.

Please just learn to develop the existing Trac instead, and work
together with existing community. You will be able to accomplish
much more with much less effort.

I'll echo Ethan's question: Have you already looked into my
suggestion to extend the existing permission system?

I did, but I would still need to hit against the database, and some other things.

​​

I think that would be a pretty simple solution to your problem (I'd
estimate one week if you can read and write Python) and I imagine
that your code would inspire other cool new ideas for how Trac
permissions can be used.

 

​Its not only about my problem, my idea is to extend ​trac to be easier to extend, or even easier to hack the core without having to understand that hard how things in a database are managed.

In addition to this, I have asked in #django about any way to implement the component manager, which I think its the greatest thing I have seen in last months (appart from the way to authz), and I have found a guy very interested on your component manager, and talking to him, he has given me an idea about how to create it.

-------------------

Well, TBH I haven't done or even used something like that before, but I think it'd be possibe to create an app and then subclass (child module) the rest of the project's apps, so the modular app would handle all the "plugins".

-------------------

I have found the django-wiki app[1], which has plugins in it, with something very similar to what he suggested.

​

​I am going to try it ​with my team, as I think its a very great idea, and will have a great profit in future developing if I finally succeed. If I finally work out some way to have all trac, with its key features ported, and succeed on it, I will come back to report.

​Take into account that if I use django, I would be able to use all the facilities that framework gives, referring specifically to DAL, urls, templating, etc.​

Javier Domingo

​[1] django-wiki: https://github.com/benjaoming/django-wiki​

[Peter Stuge]

Javier Domingo wrote:
> > I'll echo Ethan's question: Have you already looked into my
> > suggestion to extend the existing permission system?
>
> I did, but I would still need to hit against the database,
> and some other things.

Be specific. What is the problem?

It just sounds like you are afraid of (unfamiliar with?) databases.

> easier to hack the core without having to understand

This sentence perfectly sums up *everything* that is wrong with
hipster programmers.

How about FIRST understanding, and THEN hacking the core?

No - it is critical to avoid understanding at ALL cost! Even if
it means rewriting a mature application using a peppy framework.

> I have asked in #django

Your method is mindblowing. Did you consider that #django might be biased?

> I have found a guy very interested on your component manager, and
> talking to him, he has given me an idea about how to create it.

..except.. IT ALREADY EXISTS!

Stop wasting time and REUSE what is already there.

> I am going to try it with my team, as I think its a very great idea,

I think it is an absolutely horrible idea and waste of precious life.


Please please just do something useful with the existing Trac instead.


//Peter

[Noah Kantrowitz]

On Mar 5, 2013, at 10:18 AM, Peter Stuge wrote:

> Javier Domingo wrote:
>>> I'll echo Ethan's question: Have you already looked into my
>>> suggestion to extend the existing permission system?
>>
>> I did, but I would still need to hit against the database,
>> and some other things.
>
> Be specific. What is the problem?
>
> It just sounds like you are afraid of (unfamiliar with?) databases.
>
>
>> easier to hack the core without having to understand
>
> This sentence perfectly sums up *everything* that is wrong with
> hipster programmers.
>
> How about FIRST understanding, and THEN hacking the core?
>
> No - it is critical to avoid understanding at ALL cost! Even if
> it means rewriting a mature application using a peppy framework.
>
>
>> I have asked in #django
>
> Your method is mindblowing. Did you consider that #django might be biased?
>
>
>> I have found a guy very interested on your component manager, and
>> talking to him, he has given me an idea about how to create it.
>
> ..except.. IT ALREADY EXISTS!
>
> Stop wasting time and REUSE what is already there.

If you cannot be respectful of others on this list, please excuse yourself from the discussion.

--Noah

[Peter Stuge]

Noah Kantrowitz wrote:
> >>> I'll echo Ethan's question: Have you already looked into my
> >>> suggestion to extend the existing permission system?
> >>
> >> I did, but I would still need to hit against the database,
> >> and some other things.
> >
> > Be specific. What is the problem?

I hope that this will be answered, so that everyone can understand
the concrete issues.

> > How about FIRST understanding, and THEN hacking the core?

..

> > Stop wasting time and REUSE what is already there.
>
> If you cannot be respectful of others on this list, please excuse
> yourself from the discussion.

Of course I respect people, but surely you don't expect me to respect
what I think is a bad idea?

I also hope that you agree with me that understanding->hacking is the
only really desirable method, because after the initial threshold
efficiency usually rises quickly.


//Peter

[Noah Kantrowitz]

On Mar 5, 2013, at 10:52 AM, Peter Stuge wrote:

> Noah Kantrowitz wrote:
>>>>> I'll echo Ethan's question: Have you already looked into my
>>>>> suggestion to extend the existing permission system?
>>>>
>>>> I did, but I would still need to hit against the database,
>>>> and some other things.
>>>
>>> Be specific. What is the problem?
>
> I hope that this will be answered, so that everyone can understand
> the concrete issues.
>
>
>>> How about FIRST understanding, and THEN hacking the core?
> ..
>>> Stop wasting time and REUSE what is already there.
>>
>> If you cannot be respectful of others on this list, please excuse
>> yourself from the discussion.
>
> Of course I respect people, but surely you don't expect me to respect
> what I think is a bad idea?
>

Yes, we do. You do not have to agree with it, but you must respect it and conduct yourself professionally if you wish to remain a part of this conversation.

--Noah

[Javier Domingo]

2013/3/5 Peter Stuge <pe...@stuge.se>

Javier Domingo wrote:
> > I'll echo Ethan's question: Have you already looked into my
> > suggestion to extend the existing permission system?
>

> I did, but I would still need to hit against the database,
> and some other things.

Be specific. What is the problem?

It just sounds like you are afraid of (unfamiliar with?) databases.

No, just that I don't like having to make the queries specific to db.

​​

> easier to hack the core without having to understand

This sentence perfectly sums up *everything* that is wrong with
hipster programmers.

How about FIRST understanding, and THEN hacking the core?

I understood the code, what I meant is that I don't like to have to struggle my head onto the code. I would use perl if I would like to do so. I am using python because it is easier to read on a simple glance.

​​

No - it is critical to avoid understanding at ALL cost! Even if
it means rewriting a mature application using a peppy framework.

I understand that trac is a mature, but that is not a reason for not upgrading it.

In the same line of thinking, you should be using a plaintext database, as that would work too, you should use a non-preemtive kernel (for example), or develop web apps in asm.

​​

> I have asked in #django

Your method is mindblowing. Did you consider that #django might be biased?

Well, I don't see why it should be biased​​, as I just asked for a way of implementing a component manager in django, and he suggested the same solution trac is using.

> I have found a guy very interested on your component manager, and
> talking to him, he has given me an idea about how to create it.

..except.. IT ALREADY EXISTS!

Stop wasting time and REUSE what is already there.

That is my idea, I plan to use a DAL that is already there, and many more things.

​​

> I am going to try it with my team, as I think its a very great idea,

I think it is an absolutely horrible idea and waste of precious life.

Ok​​

Please please just do something useful with the existing Trac instead.

Well, that is the first thing I planned to do, but I see no bad on trying to experiment with the migration

​​

//Peter

[Javier Domingo]

> Be specific. What is the problem?

I hope that this will be answered, so that everyone can understand
the concrete issues.

I forgot to answer ^^!, the problem is that if I want to reimplement my own ticketing system, using the existing one as a base, I will have to hit against a lot of code that is hitting against the db, another part will have mixed authz and workflow.

Just using a DAL would simplify a lot the model.py (in tickets i.e.) 's code, but I was proposing going one step ahead and converting trac in a django app. I just wanted to know whether if this had been recently discussed, and which ideas trac's devs had.

This would allow someone to integrate the tracker with much a lot of webapps, which I think its great. I love doing such experiments.

[Peter Stuge]

Javier Domingo wrote:
> > > Be specific. What is the problem?
> >
> > I hope that this will be answered, so that everyone can understand
> > the concrete issues.
>
> I forgot to answer ^^!, the problem is that if I want to reimplement
> my own ticketing system, using the existing one as a base

But that wasn't what I suggested..

I suggested to implement dynamic permissions while staying with the
existing ticketing system unchanged for ticket contents.

The only code you would need to write is for permission decisions,
the result of your code within a request would be one or more
permissions set, and the regular permissions system allows or
disallows access based on that.

Of course you would still need to do database things in order to make
those permission decisions, but aren't there also some things in Trac
to help with that?

In the end, if you keep your SQL simple enough it just works<tm>
everywhere.


//Peter

[Dirk Stöcker]

Respect needs to be earned and is not there automatically. And I don't see
anything to be respected yet. Peter is perfectly right, that starting to
rework a mature and existing system without real need and without
downwards compatibility is a dumb idea and wasted life time.

And people saying that all the existing stuff on trac-hacks and elsewhere
isn't really needed and can be dropped are insane. One of my Trac
installations has 25 own site specific extensions and 50KB of code. I
guess many other installations have a lot more specific stuff.

If Javier really wants to improve things, he may fix issues, improve
existing code. And if there is a big need for database abstraction, then
he can start make patches and patches and step by step integrate an
abstraction layer. Thought I don't expect that to happen, because in this
case he would already have started sending patches. The people who start
with talking never get valuable contributors in my experience. Thought I
would be happy to be wrong for the first time. We'll see.

Ciao
--
http://www.dstoecker.eu/ (PGP key available)

[Javier Domingo]

2013/3/5 Dirk Stöcker <tr...@dstoecker.de>

On Tue, 5 Mar 2013, Noah Kantrowitz wrote:

On Mar 5, 2013, at 10:18 AM, Peter Stuge wrote:

Stop wasting time and REUSE what is already there.

If you cannot be respectful of others on this list, please excuse yourself from the discussion.

Respect needs to be earned and is not there automatically. And I don't see anything to be respected yet. Peter is perfectly right, that starting to rework a mature and existing system without real need and without downwards compatibility is a dumb idea and wasted life time.

​Respect has to meanings, at least in spanish. It means that the others have you (or your opinion) in esteem or regard, and it also means that the others don't attack you, or try to impose their opinions on you. I hope you only mean that the first one is the one to be gained, as the second one is part of a society.

Returning to the subject, your opinion is that doing something that at first glance is not needed is "a dumb idea and wasted life time", which may not be the reality, as that is your opinion. If you impose your opinion as an universal truth, then that goes against the second definition of respect.

And people saying that all the existing stuff on trac-hacks and elsewhere isn't really needed and can be dropped are insane. One of my Trac installations has 25 own site specific extensions and 50KB of code. I guess many other installations have a lot more specific stuff.

​I haven't say anything on that, but I think that having a lot of plugins for your app is not a reason for not upgrading/rewriting, is a reason for giving backwards compability.

​

If Javier really wants to improve things, he may fix issues, improve existing code. And if there is a big need for database abstraction, then he can start make patches and patches and step by step integrate an abstraction layer. Thought I don't expect that to happen, because in this case he would already have started sending patches.

Well, I have sometimes sent patches in other apps I found, but I have usually encountered that talking about what you are going to do before you do it, sometimes helps saving work. If you see the OP and the title, you will see that I was asking​​ whether if a DAL or something bigger than a DAL was going to be used or was planned to be used.

Later on, with the replies I saw, I started to develop the idea of rewriting completely trac.

The people who start with talking never get valuable contributors in my experience. Thought I would be happy to be wrong for the first time. We'll see.

​

​I usually report many bugs of apps, but the times I find a solution to a bug, that works for me, I usually send a patch. I can't ​study and correct all the bugs I see in all the apps I see.

I won't take a responsibility I can't assure I will meet, but I will try to develop trac into the django framework as an app, and will write here back here if I succeed.

​My intention wasn't to get ​any funding, just to talk about my thoughts and what I considered would be interesting. Anyway, I hope it is not wasted time,

Javier Domingo

[Noah Kantrowitz]

On Mar 5, 2013, at 11:17 AM, Dirk Stöcker wrote:

> On Tue, 5 Mar 2013, Noah Kantrowitz wrote:
>
>> On Mar 5, 2013, at 10:18 AM, Peter Stuge wrote:
>>> Stop wasting time and REUSE what is already there.
>>
>> If you cannot be respectful of others on this list, please excuse yourself from the discussion.
>
> Respect needs to be earned and is not there automatically. And I don't see anything to be respected yet. Peter is perfectly right, that starting to rework a mature and existing system without real need and without downwards compatibility is a dumb idea and wasted life time.
>

This is an unacceptable way to conduct yourself in the open source community and in the Trac community. No one here is dumb, no ideas are dumb, and no one is wasting time. You clearly disagree with Javier and thats fine (and on a personal note, so do I, I think it would be an unwise addition to Trac because the benefits wouldn't outweigh the increase in code complexity and time required to execute it), however you must express this in a way that shows that Javier is a fellow contributor and your conduct should be as it is with any other coworker or compatriot in a professional setting.

> And people saying that all the existing stuff on trac-hacks and elsewhere isn't really needed and can be dropped are insane. One of my Trac installations has 25 own site specific extensions and 50KB of code. I guess many other installations have a lot more specific stuff
>

> If Javier really wants to improve things, he may fix issues, improve existing code. And if there is a big need for database abstraction, then he can start make patches and patches and step by step integrate an abstraction layer. Thought I don't expect that to happen, because in this case he would already have started sending patches. The people who start with talking never get valuable contributors in my experience. Thought I would be happy to be wrong for the first time. We'll see.

Thats a totally reasonable point of view, and again one I share, and expressed calmly and professionally. Lets start from here; Javier maybe you could convert a small portion of the Trac codebase in your own fork to demonstrate what you think the benefits will be?

--Noah

[Javier Domingo]

Thats a totally reasonable point of view, and again one I share, and expressed calmly and professionally. Lets start from here; Javier maybe you could convert a small portion of the Trac codebase in your own fork to demonstrate what you think the benefits will be?

--Noah

​Well, that is my idea, ​I will try to implement a little part of trac in the framework, and will come back to report.

​Javier Domingo​

[Dirk Stöcker]

On Tue, 5 Mar 2013, Noah Kantrowitz wrote:

> No one here is dumb,
> no ideas are dumb,
> and no one is wasting time.

All three statements are wrong. Sorry, but polictical correctness does not
make the world different. It takes a lot to let me say a special person is
dumb, but ideas are dumb and time is wasted a lot. Understanding that
helps a lot to reduce it.

Read e.g. http://www.zdziarski.com/blog/?p=66 to understand that not every
contribution to OpenSource really is valuable and not everyone helpful.

See section "The Philosopher" and next one for reference to this thread.

> You clearly disagree with Javier

Actually that is untrue as well. I have no strong opinion regarding that
topic. I usually never have even for my own projects. I only analyze the
postings here and the related coding efforts and for me this looks
like "not worth the effort". If a suggestion to change central components
comes from a valued contributor it is usually worth discussion, from a
newbie usually not :-) A newbie in most cases can't understand what
changing a core component means really, even if he is a very good
long-time programmer.

> ... Javier is a fellow contributor ...

He may be. Possibly. In future.

I wish him all the best, but changing core components is usually not a
good start.

[Peter Stuge]

Javier Domingo wrote:
> I won't take a responsibility I can't assure I will meet, but I will
> try to develop trac into the django framework as an app, and will
> write here back here if I succeed.

I wish that you would spend that time on the existing Trac instead.

That said, happy hacking!


//Peter

[Olemis Lang]

cross-posting to blodhound-dev ML fyi . Please follow in trac-dev@...

On 3/5/13, Leho Kraav <le...@kraav.com> wrote:
>
>
>> + 1 for creating/adding a
>> DAL to Trac
>>

+1

>> I could imagine that doing a fundraising for the goal could raise some
>> money
>> for solving the human ressource problem.
>>

Just count me in
:)

IMO the following two major scenarios would have been much easier to
handle if an ORM would be handy :

1. mult-product support
2. run Trac with NoSQL DBs (e.g. MongoDB , CouchDB, ... )

so IMO it's about time to get rid of the SQL dependency .

JFTR we are using a kind of hand made ORM in Bloodhound and there's
also a way to make plugins use SQLAlchemy (plugin available @ t.h.o) .
We didn't move forward to using ORMs , DAO or such kinds of DAL
because Trac didn't in first place and it was very important for us
(at the time) to keep our core compatible with Trac's . So we
translated SQL queries under the hood , and so far looking good ...
;)

>>
> Maybe. Noone has attempted that yet.

We have . Before starting Bloodhound multi-product support we raised a
request in trac-dev (just take a look in ML archives if not already
deleted) focusing on two options :

1. using DAO pattern
2. use ORM technologies

there was no actual useful response from trac-dev , so we decided to
go our own way

[...]

> But Hans, which direction are you preferring. Porting Trac main modules ->
> let's say Django, or hack Trac's core framework up towards the giants.
>

I'd advocate using the second . Let's just choose the ORM ;)

> My spidey sense is telling me world doesn't really need yet another web
> framework. I think it would make (at least for business) sense to port the
> modules on top of bigger frameworks.
>

-1
The foundations of Trac are rock solid .

--
Regards,

Olemis.

Apache™ Bloodhound contributor
http://issues.apache.org/bloodhound

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:

[Olemis Lang]

On 3/5/13, Leho Kraav <le...@kraav.com> wrote:

> On Monday, March 4, 2013 3:20:28 PM UTC+2, Javier Domingo wrote:
>>
>> Hi,

:)

>> I am looking into trac's code, and I have found that there is no DAL.
>>
>> It has a very flexible database design, but it is not an efficient design
>> for a database that will have lots of queries.
>>

Javier , I am 80% in favor of doing this .

please read my previous message ...

>> I have found that it uses hard-coded queries (which is difficult to
>> understand how it works), text as primary keys, etc.
>>
>> I would like to ask why doesn't trac use django (with south for db
>> versioning).
>>

... because Django came into the scene afterwards , they even use Trac
as issue tracker afaicr ;)

>> I plan to develop a parallel ticket system as a plugin (as I mentioned in
>> previous threads) using as base the original trac.ticket structure, using
>> django as a DAL and maintaining the interfaces.
>>

see SQL Alchemy Trac bridge , and do a similar thing for Trac .

[...]
>
> Hi Javier
>
> Trac is almost 10 years old. Django (or anything else, Pyramid etc) wasn't
> nearly the quality players they are today. Trac is also a web application
> framework on its own. It has therefore never needed an outside framework to
> perform its job.
>

+1

[Olemis Lang]

On 3/5/13, Olemis Lang <ole...@gmail.com> wrote:
> On 3/5/13, Leho Kraav <le...@kraav.com> wrote:
>> On Monday, March 4, 2013 3:20:28 PM UTC+2, Javier Domingo wrote:
>>>

[...]

>
> see SQL Alchemy Trac bridge , and do a similar thing for Trac .
>

oops ! ... and do a similar thing for **Django ORM** .

[Christian Boos]

On 3/5/2013 6:13 PM, Peter Stuge wrote:
> Javier Domingo wrote:
>> I think that going django would mean a complete remake of trac, but
>> would simplify a lot trac deployment (in for example appengine)
>
> Stop the madness.
>
> Trac has existed for 10+ years. You're a student who doesn't really
> know Python (you said).
>
> It makes no sense whatsoever for you to rewrite Trac using hipster
> frameworks!

Well, something like that has been attempted already, also by people
wanting to use Trac for supporting university projects. Greg Wilson and
his student teams fro U. of Toronto started DrProject, a fork of Trac,
in 2006 (IIRC) aiming at a simplified interface and multi-project
support. At some point they decided to rewrite this on top of Django:
the Basie project was born. Then apparently they decided to move from
that to Pinax (1). That might be already closer to what you want to
achieve...

>
> You would be completely alone working on that, it would take an
> enormous amount of time because you have to learn so many things,
> and in the end it's not likely going to be any better than what is
> already there.

Switching to a new cool "technology" can feel attractive and there might
be some advantages in the long run... or not, it's always hard to tell
beforehand. Witness our Genshi migration, which was a very long trip,
with lots of ups and downs. It's true that Peewee looks nice (thanks for
the pointer, Hans-Peter!), but using it or something similar is not a
light decision to take (a switch to SqlAlchemy was suggested or tried
several times since 2006, it didn't happen, maybe for good reasons). And
I tend to agree with Peter, even if such a switch would be successful,
would Trac really be significantly better after? I think that this
amount of effort and time would be better spend on rethinking the
*model* itself, rather than focusing on the cosmetics of using the model
we have.

>
> Please just learn to develop the existing Trac instead, and work
> together with existing community. You will be able to accomplish
> much more with much less effort.

Trac, or anything closer to what you're aiming at (check the Pinax thing
mentioned above, someone suggested you have a look at BloodHound, and if
you're not too deep already in Python, you also have
Redmine/ChiliProject to look at). My point is that if you realize that
the Trac fundamentals don't suit you or your purpose, maybe you would be
better off to start with something that fits better, as there's no lack
of alternative models. This is not to say that everything is perfect in
Trac, of course...

-- Christian

(1) http://third-bit.com/blog/archives/4079.html

[Leho Kraav]

Warning: possible OT stuff below. Javier seems to have found his path
already based on his last messages. I hope mainly he as the OP doesn't mind.

On 05.03.2013 23:47, Olemis Lang wrote:
>
> I'd advocate using the second . Let's just choose the ORM ;)
>
>> My spidey sense is telling me world doesn't really need yet another
>> web framework. I think it would make (at least for business) sense
>> to port the modules on top of bigger frameworks.
>>
>
> -1 The foundations of Trac are rock solid.

I don't doubt this. My only concern is the size of a general purpose
plugin ecosystem around Trac today *and in foreseeable future*. Compared
to WordPress, which is what I'm working with currently day to day, we're
talking a drop in a bucket. And WP started as a simple blog engine, just
like Trac started as a simple issue tracker. Today WP a really solid
general purpose app platform with *massive* code drop-in and reuse
options via https://wordpress.org/extend/plugins/ (23,824 plugins,
422,393,833 downloads, and counting) + themes + countless commercial
plugin product offers.

Means you get various real business done on it with a fraction of exotic
platform costs, aside from having to go through discovering some
terrible PHP programmers and alternatives to their packages. But that's
peanuts compared to writing stuff from scratch.

To me, all this DAL or no-DAL, Django or no-Django etc is connected this
way. At the very basic level, Trac does some very important things for
me very well. Which is also probably why we're all sitting here and
don't completely move on to something else.

It'd be really, really nice (dare I say profitable) to get even more out
of it for everyone.

So in my case, it either means separate interconnected systems (leave
Trac as it is, build other stuff in some other framework, connect via
RPC or whatever) or writing a looott of new Trac-stuff from scratch.
Trac shopping cart, anybody? Real trac-hacks marketplace, anybody? Just
a few examples that interest me on top of the issue tracking awesomeness
we have today :)

But when I'm looking at how long it takes for a *whole team, focused* to
develop a solid e-commerce solution

https://github.com/woothemes/woocommerce (WP)
https://github.com/diefenbach/django-lfs (Django)
http://www.satchmoproject.com/ (Django)

these take about 2 years just to reach a solid 1.0 + another year for 2.0.

It just feels like we're all going to grow grey and old, before anything
reasonable could be achieved going in the "write new implementations of
world on top of Trac-as-web-framework" direction. Maybe that's overly
pessimistic, but software is hard (ironic eh) so the negative scenario
is always significantly more likely.

E-commerce is mainly as an example here; good or bad, maybe irrelevant.
It's just what I'm working with day to day, atm, and therefore thinking
about. Money makes stuff go around and I can't help feeling that it's
strange there seems to be almost no commercially supported Trac plugins
available. Anything besides Agilo out there?

Not to say a Trac marketplace couldn't be hosted on something else, but
trac-hacks is the center atm and we all see how painfully long the
development process is with it. If it had more of a selection of solid
drop-in plugins it could benefit from (even if commercial), maybe the
0.12 rewrite would've been done long ago and we'd all get to enjoy the
fruits before we have grey beards.

Maybe there would be more if there was more of a feel of being able to
build a business on Trac?

But there's no wide choice or business feel because... *maybe* partly
because there's no DAL, to start. New blood doesn't easily flow into the
ecosystem, just hacking along on a variety of world implementations,
significantly lifting the "created a plugin" count and with that comes
rise in "widely used, successful, maintained plugin" % chart. Obviously
it's also because Trac has never been advertised as a general purpose
web framework. But even if people discover this capability, they
probably pick an ORM-ified alternative because it's just simpler to try
stuff against. At the end of the day, it's all a % game.

[Javier Domingo]

Well, something like that has been attempted already, also by people
wanting to use Trac for supporting university projects. Greg Wilson and
his student teams fro U. of Toronto started DrProject, a fork of Trac,
in 2006 (IIRC) aiming at a simplified interface and multi-project
support. At some point they decided to rewrite this on top of Django:
the Basie project was born. Then apparently they decided to move from
that to Pinax (1). That might be already closer to what you want to
achieve...

​I have found a mirror that hasn't come down on launchpad[1] and having a look on it is not the thing I want to do. Anyway, thank you a lot for the tip.

Javier Domingo

​[1] Basie mirror: http://bazaar.launchpad.net/~vcs-imports/basie/trunk/files/head:/​

[Leho Kraav]

On 06.03.2013 02:05, Christian Boos wrote:
>
> support. At some point they decided to rewrite this on top of Django:
> the Basie project was born. Then apparently they decided to move from
> that to Pinax (1). That might be already closer to what you want to
> achieve...

> ...

> Trac, or anything closer to what you're aiming at (check the Pinax thing
> mentioned above, someone suggested you have a look at BloodHound, and if

>...

This Pinax thing is much interesting, thanks Christian

[Javier Domingo]

This Pinax thing is much interesting, thanks Christian

​I found this about pinax: https://github.com/pinax/pinax then it really is something similar to what I plan to do.

[Brettschneider Falk]

Dirk Stöcker wrote:
> If Javier really wants to improve things, he may fix issues, improve
> existing code. And if there is a big need for database abstraction, then
> he can start make patches and patches and step by step integrate an
> abstraction layer. Thought I don't expect that to happen, because in this
> case he would already have started sending patches. The people who start
> with talking never get valuable contributors in my experience. Thought I
> would be happy to be wrong for the first time. We'll see.

Off-topic: Although I understand your pessimistic view a bit, it can also be the case that people simply want to get positive feedback like "yes, try it, maybe we'll accept such patch but pay attention to ..." before they start to spend time with coding with the risk of getting rejected afterwards. That's a problem with established long-time projects where the number of *new* core developers is getting rare, the fear to touch the sacred cow. :)

CU, F@lk

----
R&D Software
Baumer Optronic GmbH
www.baumer.com




Geschäftsführer: Dr. Albert Schmidt· Dr. Oliver Vietze
Sitz der Gesellschaft: Radeberg
Amtsgericht Dresden: HRB 15379
Ust. ID: DE 189714583

[Peter Stuge]

Brettschneider Falk wrote:
> the fear to touch the sacred cow. :)

I think it's great when someone wants to touch the cow, but I also
think that there is a difference between touching the cow and
exchanging the cow for a yak. ;)


//Peter

[Olemis Lang]

On 3/6/13, Leho Kraav <le...@kraav.com> wrote:
[...]

> On 05.03.2013 23:47, Olemis Lang wrote:
>>
>> I'd advocate using the second . Let's just choose the ORM ;)
>>
>>> My spidey sense is telling me world doesn't really need yet another
>>> web framework. I think it would make (at least for business) sense
>>> to port the modules on top of bigger frameworks.
>>>
>>
>> -1 The foundations of Trac are rock solid.
>
> I don't doubt this. My only concern is the size of a general purpose
> plugin ecosystem around Trac today *and in foreseeable future*. Compared
> to WordPress, which is what I'm working with currently day to day, we're
> talking a drop in a bucket. And WP started as a simple blog engine, just
> like Trac started as a simple issue tracker. Today WP a really solid
> general purpose app platform with *massive* code drop-in and reuse
> options via https://wordpress.org/extend/plugins/ (23,824 plugins,
> 422,393,833 downloads, and counting) + themes + countless commercial
> plugin product offers.
>

those are facts ... why bother /me saying *agreed*

[...]

>
> To me, all this DAL or no-DAL, Django or no-Django etc is connected this
> way. At the very basic level, Trac does some very important things for
> me very well. Which is also probably why we're all sitting here and
> don't completely move on to something else.
>

To me I see the subject from a more *practical* perspective ...

> So in my case, it either means separate interconnected systems (leave
> Trac as it is, build other stuff in some other framework, connect via
> RPC or whatever) or writing a looott of new Trac-stuff from scratch.

... which is , in either case we'll need to improve DB code ... say
DBAL ... so we better start doing so tomorrow so that we can see how
it goes after one ? two ? three ? ... months .

> Trac shopping cart, anybody?



> Real trac-hacks marketplace, anybody?

IMHO that's another debate ... and does not depend *only* on the good
faith of developers doing something and people tracking but consumers
and marketing , and ... I'm just saying it's another subject .
;)

> Just
> a few examples that interest me on top of the issue tracking awesomeness
> we have today :)
>

Let's see if you have a WSGI web app doing all that then what stops
you from embedding inside Trac ? What prevents somebody to tightly
integrate Trac and e.g. Django at a very low level thus providing
seamless integration , deployment and management ? I know the first
part of the answer : the limitations of Trac database layer . Let's
fix it , and focus on the rest once that will be done .

Summarizing , IMO there many approaches to integrate two systems and
make them interoperate .

[...]

>
> It just feels like we're all going to grow grey and old, before anything
> reasonable could be achieved going in the "write new implementations of
> world on top of Trac-as-web-framework" direction.

... but maybe not on the direction of «improve Trac integration with
established & popular technologies»

> Maybe that's overly
> pessimistic, but software is hard (ironic eh) so the negative scenario
> is always significantly more likely.
>

you definitely have a point . What shall we do ? I have the feeling
that you'll always end up questioning DB and SQL . I'd rather say ,
let's do that and find ways afterwards to either :

1. make current Trac a better Trac
2. build something else from scratch ... well ... from DBAL layer up ;)

[...]

> Maybe there would be more if there was more of a feel of being able to
> build a business on Trac?
>
> But there's no wide choice or business feel because... *maybe* partly
> because there's no DAL, to start.

+1 ... not exactly because of that solely reason , but because of what
it implies : a strong limitation to get a lot of things done , leaving
some deployments out of the equation .

[...]

[Olemis Lang]

On 3/6/13, Leho Kraav <le...@kraav.com> wrote:
[...]

> On 05.03.2013 23:47, Olemis Lang wrote:
>>
>> I'd advocate using the second . Let's just choose the ORM ;)
>>
>>> My spidey sense is telling me world doesn't really need yet another
>>> web framework. I think it would make (at least for business) sense
>>> to port the modules on top of bigger frameworks.
>>>
>>
>> -1 The foundations of Trac are rock solid.
>
> I don't doubt this. My only concern is the size of a general purpose
> plugin ecosystem around Trac today *and in foreseeable future*. Compared
> to WordPress, which is what I'm working with currently day to day, we're
> talking a drop in a bucket. And WP started as a simple blog engine, just
> like Trac started as a simple issue tracker. Today WP a really solid
> general purpose app platform with *massive* code drop-in and reuse
> options via https://wordpress.org/extend/plugins/ (23,824 plugins,
> 422,393,833 downloads, and counting) + themes + countless commercial
> plugin product offers.
>

those are facts ... why bother /me saying *agreed*

[...]
>

> To me, all this DAL or no-DAL, Django or no-Django etc is connected this
> way. At the very basic level, Trac does some very important things for
> me very well. Which is also probably why we're all sitting here and
> don't completely move on to something else.
>

To me I see the subject from a more *practical* perspective ...

> So in my case, it either means separate interconnected systems (leave
> Trac as it is, build other stuff in some other framework, connect via
> RPC or whatever) or writing a looott of new Trac-stuff from scratch.

... which is , in either case we'll need to improve DB code ... say
DBAL ... so we better start doing so tomorrow so that we can see how
it goes after one ? two ? three ? ... months .

> Trac shopping cart, anybody?

:-$

> Real trac-hacks marketplace, anybody?

IMHO that's another debate ... and does not depend *only* on the good
faith of developers doing something and people tracking but consumers
and marketing , and ... I'm just saying it's another subject .
;)

> Just
> a few examples that interest me on top of the issue tracking awesomeness
> we have today :)
>

Let's see if you have a WSGI web app doing all that then what stops
you from embedding inside Trac ? What prevents somebody to tightly
integrate Trac and e.g. Django at a very low level thus providing
seamless integration , deployment and management ? I know the first
part of the answer : the limitations of Trac database layer . Let's
fix it , and focus on the rest once that will be done .

Let's put things in context ... I'd like to be able to install
wordpress themes and make them work ootb to change Trac appearance .
As long as it's technically feasible, what prevents somebody from
writing a generic TracWordpressThemeIntegration plugin and make all
such skins fall like dominoes ?

Summarizing , IMO there many approaches to integrate two systems and
make them interoperate .

[...]
>

> It just feels like we're all going to grow grey and old, before anything
> reasonable could be achieved going in the "write new implementations of
> world on top of Trac-as-web-framework" direction.

... but maybe not on the direction of «improve Trac integration with
established & popular technologies»

> Maybe that's overly
> pessimistic, but software is hard (ironic eh) so the negative scenario
> is always significantly more likely.
>

you definitely have a point . What shall we do ? I have the feeling
that you'll always end up questioning DB and SQL . I'd rather say ,
let's do that and find ways afterwards to either :

1. make current Trac a better Trac
2. build something else from scratch ... well ... from DBAL layer up ;)

[...]

> Maybe there would be more if there was more of a feel of being able to
> build a business on Trac?
>
> But there's no wide choice or business feel because... *maybe* partly
> because there's no DAL, to start.

+1 ... not exactly because of that solely reason , but because of what
it implies : a strong limitation to get a lot of things done , leaving
some deployments out of the equation .

[...]