Diem 5.4 new features and enhancements, still work in progress

95 views
Skip to first unread message

Stéphane

unread,
Jan 8, 2011, 12:25:03 PM1/8/11
to diem-...@googlegroups.com
Hello Community,

I here will take some time to explain you what has been done for now on the 5.4 branch.


  • Diem Doctrine Admin module generator :
    1. Adding a buildObjectQuery() in the dmAdminBaseGeneratedActions, so you can fine tune the building of the query to retrieve an object.
      1. This method will $table->joinAll($query) with I18n if any. This improves the db querying with fewer queries.
      2. You can overload this method within your actions !
      3. This is only used by the edit action, not the list action (which uses the buildQuery() method)
    2. modified the getObject() method so if no 'pk' is found in the request, it will search for 'id' instead (using $table->getIdentifier())
      1. Usefull when you create new actions and add them in object_actions or actions in generator.yml
    3. Within lists and edit actions, the links to actions are shown according to user credentials (action, mixed and record are managed)

  • Diem Doctrine Form generator :
    1. Better handling of relations of type One To Many and One To One within templates

  • Diem Doctrine Form :
    1. Adding a way to specify the order of saving of embedded forms by using the $form->setEmbeddedFormSavingTime() This is really usefull when you embed a form representing a Doctrine_Relation_ForeignKey, which has to be saved BEFORE the form's object.
      So you can specify if an embedded form has to be saved BEFORE or AFTER the record, so you'll not get an integrity constraint violation (as the embedded form's record will get saved after the record)

  • Diem Doctrine Admin module security strategies :
    1. There are three strategies for securing Module-Actions :
      1. action : the symfony way, nothing new, except it has been strategified so all security strategies are working the same way
      2. record : secure a module-action for a given record. The user must be associated to the corresponding DmRecordPermission
      3. mixed : a mix of action and record (usefull for listing, or avoiding the querying if user do not have the necessary dmPermission at first)
        1. Personnaly, if I want to secure a MA using record, I'll use mixed instead (so mechanism will check for user credentials before querying db)

  • Diem Doctrine Loremizer :
    1. Adding the "double" doctrine type

  • Diem Doctrine Query DQL Tokenizer :
    1. I'm using this extension https://github.com/ebihara/php-dql-tokenizer
    2. So I made dmDoctrineQuery aware of this using a condition (see here https://github.com/diem-project/diem/commit/b2223d06ea0b9f831bad91df13743536b16d0311
  • CSS modifications :
    1. Adding a z-index to the class of the resize handler for markdown editor, so it will not get over the menus
    2. Setting the max-height to 700px for the menus (I have big menus and I don't want them to be scrolled), what do you think ?
  • Docblocks here and there for better IDE auto completion
  • Diem tasks:
    1. dm:setup modified
      1. There is a new option --load-doctrine-data which will add this option to the dm:data task call
        1. It will load doctrine data fixtures
      2. There is a new option --clear-tables which will drop all the tables of the db, instead of dropping the db itself. This is explained in its own task description following.
    2. new DmPermission generator task dm:generate-permissions-fixtures (didn't find something longer -_-)
      1. It will parse all the modules.yml files and search for credentials:, and generate fixture files for these permissions.
      2. I plan to create classes for these too (which will make it faster at loading)
    3. dm:data modified:
      1. There is a new option --load-doctrine-data that will call doctrine:data-load
      2. This is used by dm:setup too
    4. new dm:drop-tables task :
      1. I have my db loaded within a tmpfs (ramdisk), so I can drop my database because mysql will delete the directory, thus deleting the symlink, and recreate it, thus the database will no more be in my tmpfs. That's why I need a drop tables instead of a drop db.
  • Admin menus shows only user-accessible menus using defined credentials or global module credentials
  • Diem Admin module generator is now I18n catalogue "aware" : https://github.com/diem-project/diem/commit/dba156e1b3cdbf165d4645c2bfb3f15d3f297f4f
    • You can specify the catalogue to use in the modules.yml using i18n_catalogue, and all the templates generated in cache/ will use this catalogue instead of the dm or messages catalogues.
    • Modified the dmAdminModelGeneratorHelper->getI18nCatalogue() to fallback to 'dm' if none specified in modules.yml
My future plans :

  1. XHProf embedding : Diem already embed APC pages, so I thought embedding XHProf would be great too ! As soon as the XHProf module is enabled, it will get embedded, and you'll be able to get a link to the "run" to see the profiling of the run (check XHProf documentation for this)
  2. Creating a dmDev: task namespace with some great tasks like "creating vhost", "creating fake dns", "creating tmpfs and put db in it so you'll get faster dev times and testing, and still using innodb", etc.


I might have forgotten some things, but you'll be able to find all commits in github :)


Regards,
Before Printing, Think about Your Environmental Responsibility!
Avant d'Imprimer, Pensez à Votre Responsabilitée Environnementale!

Stéphane

unread,
Jan 8, 2011, 1:15:32 PM1/8/11
to diem-...@googlegroups.com
In the things to do, I have forgotten :

  • replacing the front components for user signin, login/logout and registering with one component
  • being able to have the user interface in one language while having the content in another language (usefull for translators), this would be optional
  • paginating the Page toolbar when there is a lot of pages (showing a ...), and why not adding a module for managing pages (merging this toolbar + the reorganize action
  • paginating the *-to-many when there are too many items to show in an admin module + auto completion using jquery
  • moving to latest jquery version
  • making the filter form to be resizable (or make it auto using css ?)
  • adding the "type" to filter forms for master class (managing inheritance)
I forgot to mention some things I've done too :

  • the generated forms are now fine with inheritance, adding a "type" showing all the subtypes for the master class, plus generating good widgets for sub forms, and generating good widgets too on the master class (so master class have no more Interger widget when it is a column created by a subform, this is managed by recoding the getColumns and adding it two options parameters to include or not subclasses columns and relations)

I NEED HELP =)

Regards,

Before Printing, Think about Your Environmental Responsibility!
Avant d'Imprimer, Pensez à Votre Responsabilitée Environnementale!


h16bit

unread,
Jan 8, 2011, 11:02:07 PM1/8/11
to diem-users
Stephane, i commit my global area path many time ago(but this commit
was forgotten), in 2.0 version this patch is BC and save caching
system, so i think it can be included to new version of diem.

On 8 янв, 23:15, Stéphane <stephane.er...@gmail.com> wrote:
> In the things to do, I have forgotten :
>
>    - replacing the front components for user signin, login/logout and
>    registering with one component
>    - being able to have the user interface in one language while having the
>    content in another language (usefull for translators), this would be
>    optional
>    - paginating the Page toolbar when there is a lot of pages (showing a
>    ...), and why not adding a module for managing pages (merging this toolbar +
>    the reorganize action
>    - paginating the *-to-many when there are too many items to show in an
>    admin module + auto completion using jquery
>    - moving to latest jquery version
>    - making the filter form to be resizable (or make it auto using css ?)
>    - adding the "type" to filter forms for master class (managing
>    inheritance)
>
> I forgot to mention some things I've done too :
>
>    - the generated forms are now fine with inheritance, adding a "type"
>    showing all the subtypes for the master class, plus generating good widgets
>    for sub forms, and generating good widgets too on the master class (so
>    master class have no more Interger widget when it is a column created by a
>    subform, this is managed by recoding the getColumns and adding it two
>    options parameters to include or not subclasses columns and relations)
>
> I NEED HELP =)
>
> Regards,
>
> Before Printing, Think about Your Environmental Responsibility!
> Avant d'Imprimer, Pensez à Votre Responsabilitée Environnementale!
>
>
>
> On Sat, Jan 8, 2011 at 6:25 PM, Stéphane <stephane.er...@gmail.com> wrote:
> > Hello Community,
>
> > I here will take some time to explain you what has been done for now on the
> > 5.4 branch.
>
> >    - Diem Doctrine Admin module generator :
>
> >    1. Adding a buildObjectQuery() in the dmAdminBaseGeneratedActions, so
> >       you can fine tune the building of the query to retrieve an object.
> >          1. This method will $table->joinAll($query) with I18n if any.
> >          This improves the db querying with fewer queries.
> >          2. You can overload this method within your actions !
> >          3. This is only used by the edit action, not the list action
> >          (which uses the buildQuery() method)
> >       2. modified the getObject() method so if no 'pk' is found in the
> >       request, it will search for 'id' instead (using $table->getIdentifier())
> >          1. Usefull when you create new actions and add them in
> >          object_actions or actions in generator.yml
> >       3. Within lists and edit actions, the links to actions are shown
> >       according to user credentials (action, mixed and record are managed)
>
> >    - Diem Doctrine Form generator :
>
> >    1. Better handling of relations of type One To Many and One To One
> >       within templates
>
> >    - Diem Doctrine Form :
>
> >    1. Adding a way to specify the order of saving of embedded forms by
> >       using the $form->*setEmbeddedFormSavingTime() *This is really
> >       usefull when you embed a form representing a Doctrine_Relation_ForeignKey,
> >       which has to be saved BEFORE the form's object.
> >       So you can specify if an embedded form has to be saved BEFORE or
> >       AFTER the record, so you'll not get an integrity constraint violation (as
> >       the embedded form's record will get saved after the record)
>
> >    - Diem Doctrine Admin module security strategies :
>
> >    1. There are three strategies for securing Module-Actions :
> >          1. action : the symfony way, nothing new, except it has been
> >          strategified so all security strategies are working the same way
> >          2. record : secure a module-action for a given record. The user
> >          must be associated to the corresponding DmRecordPermission
> >          3. mixed : a mix of action and record (usefull for listing, or
> >          avoiding the querying if user do not have the necessary dmPermission at
> >          first)
> >             1. Personnaly, if I want to secure a MA using record, I'll use
> >             mixed instead (so mechanism will check for user credentials before querying
> >             db)
> >          2.
>
> >    - Diem Doctrine Loremizer :
>
> >    1. Adding the "double" doctrine type
>
> >    - Diem Doctrine Query DQL Tokenizer :
>
> >    1. I'm using this extension
> >      https://github.com/ebihara/php-dql-tokenizer<https://github.com/diem-project/diem/commit/b2223d06ea0b9f831bad91df1...>
> >       2. So I made dmDoctrineQuery aware of this using a condition (see
> >       here
> >      https://github.com/diem-project/diem/commit/b2223d06ea0b9f831bad91df1...
>
> >    - CSS modifications :
>
> >    1. Adding a z-index to the class of the resize handler for markdown
> >       editor, so it will not get over the menus
> >       2. Setting the max-height to 700px for the menus (I have big menus
> >       and I don't want them to be scrolled), what do you think ?
>
> >    - Docblocks here and there for better IDE auto completion
> >    - Diem tasks:
>
> >    1. dm:setup modified
> >          1. There is a new option --load-doctrine-data which will add this
> >          option to the dm:data task call
> >             1. It will load doctrine data fixtures
> >          2. There is a new option --clear-tables which will drop all the
> >          tables of the db, instead of dropping the db itself. This is explained in
> >          its own task description following.
> >       2. new DmPermission generator task dm:generate-permissions-fixtures
> >       (didn't find something longer -_-)
> >          1. It will parse all the modules.yml files and search for
> >          credentials:, and generate fixture files for these permissions.
> >          2. I plan to create classes for these too (which will make it
> >          faster at loading)
> >       3. dm:data modified:
> >          1. There is a new option --load-doctrine-data that will call
> >          doctrine:data-load
> >          2. This is used by dm:setup too
> >       4. new dm:drop-tables task :
> >          1. I have my db loaded within a tmpfs (ramdisk), so I can drop my
> >          database because mysql will delete the directory, thus deleting the symlink,
> >          and recreate it, thus the database will no more be in my tmpfs. That's why I
> >          need a drop tables instead of a drop db.
>
> >    - Admin menus shows only user-accessible menus using defined
> >    credentials or global module credentials
> >    - Diem Admin module generator is now I18n catalogue "aware" :
> >    https://github.com/diem-project/diem/commit/dba156e1b3cdbf165d4645c2b...
> >       - You can specify the catalogue to use in the modules.yml using
> >       i18n_catalogue, and all the templates generated in cache/ will use this
> >       catalogue instead of the dm or messages catalogues.
> >       - Modified the dmAdminModelGeneratorHelper->getI18nCatalogue() to
> >       fallback to 'dm' if none specified in modules.yml
>
> > My future plans :
>
> >    1. XHProf embedding : Diem already embed APC pages, so I thought
> >    embedding XHProf would be great too ! As soon as the XHProf module is
> >    enabled, it will get embedded, and you'll be able to get a link to the "run"
> >    to see the profiling of the run (check XHProf documentation for this)
> >    2. Creating a dmDev: task namespace with some great tasks like

h16bit

unread,
Jan 8, 2011, 11:03:15 PM1/8/11
to diem-users
And one more - i think patchs for hiding pages of some models from
admin tree can be useful too

On 8 янв, 23:15, Stéphane <stephane.er...@gmail.com> wrote:
> In the things to do, I have forgotten :
>
>    - replacing the front components for user signin, login/logout and
>    registering with one component
>    - being able to have the user interface in one language while having the
>    content in another language (usefull for translators), this would be
>    optional
>    - paginating the Page toolbar when there is a lot of pages (showing a
>    ...), and why not adding a module for managing pages (merging this toolbar +
>    the reorganize action
>    - paginating the *-to-many when there are too many items to show in an
>    admin module + auto completion using jquery
>    - moving to latest jquery version
>    - making the filter form to be resizable (or make it auto using css ?)
>    - adding the "type" to filter forms for master class (managing
>    inheritance)
>
> I forgot to mention some things I've done too :
>
>    - the generated forms are now fine with inheritance, adding a "type"
>    showing all the subtypes for the master class, plus generating good widgets
>    for sub forms, and generating good widgets too on the master class (so
>    master class have no more Interger widget when it is a column created by a
>    subform, this is managed by recoding the getColumns and adding it two
>    options parameters to include or not subclasses columns and relations)
>
> I NEED HELP =)
>
> Regards,
>
> Before Printing, Think about Your Environmental Responsibility!
> Avant d'Imprimer, Pensez à Votre Responsabilitée Environnementale!
>
>
>
> On Sat, Jan 8, 2011 at 6:25 PM, Stéphane <stephane.er...@gmail.com> wrote:
> > Hello Community,
>
> > I here will take some time to explain you what has been done for now on the
> > 5.4 branch.
>
> >    - Diem Doctrine Admin module generator :
>
> >    1. Adding a buildObjectQuery() in the dmAdminBaseGeneratedActions, so
> >       you can fine tune the building of the query to retrieve an object.
> >          1. This method will $table->joinAll($query) with I18n if any.
> >          This improves the db querying with fewer queries.
> >          2. You can overload this method within your actions !
> >          3. This is only used by the edit action, not the list action
> >          (which uses the buildQuery() method)
> >       2. modified the getObject() method so if no 'pk' is found in the
> >       request, it will search for 'id' instead (using $table->getIdentifier())
> >          1. Usefull when you create new actions and add them in
> >          object_actions or actions in generator.yml
> >       3. Within lists and edit actions, the links to actions are shown
> >       according to user credentials (action, mixed and record are managed)
>
> >    - Diem Doctrine Form generator :
>
> >    1. Better handling of relations of type One To Many and One To One
> >       within templates
>
> >    - Diem Doctrine Form :
>
> >    1. Adding a way to specify the order of saving of embedded forms by
> >       using the $form->*setEmbeddedFormSavingTime() *This is really
> >       usefull when you embed a form representing a Doctrine_Relation_ForeignKey,
> >       which has to be saved BEFORE the form's object.
> >       So you can specify if an embedded form has to be saved BEFORE or
> >       AFTER the record, so you'll not get an integrity constraint violation (as
> >       the embedded form's record will get saved after the record)
>
> >    - Diem Doctrine Admin module security strategies :
>
> >    1. There are three strategies for securing Module-Actions :
> >          1. action : the symfony way, nothing new, except it has been
> >          strategified so all security strategies are working the same way
> >          2. record : secure a module-action for a given record. The user
> >          must be associated to the corresponding DmRecordPermission
> >          3. mixed : a mix of action and record (usefull for listing, or
> >          avoiding the querying if user do not have the necessary dmPermission at
> >          first)
> >             1. Personnaly, if I want to secure a MA using record, I'll use
> >             mixed instead (so mechanism will check for user credentials before querying
> >             db)
> >          2.
>
> >    - Diem Doctrine Loremizer :
>
> >    1. Adding the "double" doctrine type
>
> >    - Diem Doctrine Query DQL Tokenizer :
>
> >    1. I'm using this extension
> >      https://github.com/ebihara/php-dql-tokenizer<https://github.com/diem-project/diem/commit/b2223d06ea0b9f831bad91df1...>
> >       2. So I made dmDoctrineQuery aware of this using a condition (see
> >       here
> >      https://github.com/diem-project/diem/commit/b2223d06ea0b9f831bad91df1...
>
> >    - CSS modifications :
>
> >    1. Adding a z-index to the class of the resize handler for markdown
> >       editor, so it will not get over the menus
> >       2. Setting the max-height to 700px for the menus (I have big menus
> >       and I don't want them to be scrolled), what do you think ?
>
> >    - Docblocks here and there for better IDE auto completion
> >    - Diem tasks:
>
> >    1. dm:setup modified
> >          1. There is a new option --load-doctrine-data which will add this
> >          option to the dm:data task call
> >             1. It will load doctrine data fixtures
> >          2. There is a new option --clear-tables which will drop all the
> >          tables of the db, instead of dropping the db itself. This is explained in
> >          its own task description following.
> >       2. new DmPermission generator task dm:generate-permissions-fixtures
> >       (didn't find something longer -_-)
> >          1. It will parse all the modules.yml files and search for
> >          credentials:, and generate fixture files for these permissions.
> >          2. I plan to create classes for these too (which will make it
> >          faster at loading)
> >       3. dm:data modified:
> >          1. There is a new option --load-doctrine-data that will call
> >          doctrine:data-load
> >          2. This is used by dm:setup too
> >       4. new dm:drop-tables task :
> >          1. I have my db loaded within a tmpfs (ramdisk), so I can drop my
> >          database because mysql will delete the directory, thus deleting the symlink,
> >          and recreate it, thus the database will no more be in my tmpfs. That's why I
> >          need a drop tables instead of a drop db.
>
> >    - Admin menus shows only user-accessible menus using defined
> >    credentials or global module credentials
> >    - Diem Admin module generator is now I18n catalogue "aware" :
> >    https://github.com/diem-project/diem/commit/dba156e1b3cdbf165d4645c2b...
> >       - You can specify the catalogue to use in the modules.yml using
> >       i18n_catalogue, and all the templates generated in cache/ will use this
> >       catalogue instead of the dm or messages catalogues.
> >       - Modified the dmAdminModelGeneratorHelper->getI18nCatalogue() to
> >       fallback to 'dm' if none specified in modules.yml
>
> > My future plans :
>
> >    1. XHProf embedding : Diem already embed APC pages, so I thought
> >    embedding XHProf would be great too ! As soon as the XHProf module is
> >    enabled, it will get embedded, and you'll be able to get a link to the "run"
> >    to see the profiling of the run (check XHProf documentation for this)
> >    2. Creating a dmDev: task namespace with some great tasks like

Stéphane

unread,
Jan 9, 2011, 11:45:19 AM1/9/11
to diem-...@googlegroups.com
Hello, 


For sure I will merge your patchs with the 5.4 ! Thank you !

I have just finished optimizing the listing and editing queries.
Now they are optimized in the sense they will join every tables needed by the view, by using the getListDisplay or getEditDisplay or whatever from the configuration object. So you get one "big" query one time and no more, except for *-to-many where the view needs to show "all" the existings records of related model (when editing).

Before, the view was making many requests to the objects, ending in many little queries, which were all related to the objects themselves (executeList it was about each row, to find each related records for each relations to be shown, executeEdit it was about the object being edited, the same thing PLUS showing all existing records from related model, so you can pick more to associate with object). I'm not talking about the other queries to show all existing records for related model as I've said.
Now it makes one big query.

About the other queries to show all existing records for related models, they still exist in executeEdit (and that's why I want to paginate this too, btw).


So please all, make the world better, update your diem 5.4 branch ;)

Regards,


Before Printing, Think about Your Environmental Responsibility!
Avant d'Imprimer, Pensez à Votre Responsabilitée Environnementale!


Thomas Ohms

unread,
Jan 10, 2011, 6:30:56 AM1/10/11
to diem-...@googlegroups.com
Stephane,

You really did a great work!! THANKS! With the new security options I can go on with dmForumPlugin for a good way of securing single posts and topics (as soon as Eric answered my open questions though). Soon this plugin's finished I'll help you with later versions.

But please don't forget the documentation of Diem's API. This is most important for all developers making it easier to enhance Diem and fix bugs. Remember my question I had on Skype? It would have saved me a lot of time if this had been documented ;)

Btw.: the idea of dropping tables instead of database has one more benefit. If a user has no right to drop databases, they still can clear the database now ;)

Regards,
Thomas

2011/1/9 Stéphane <stephan...@gmail.com>

4le...@gmail.com

unread,
Jan 10, 2011, 7:30:15 AM1/10/11
to diem-...@googlegroups.com

Hi all,

I'm also very excited about the new security enhancements, this is a long awaited feature bringing Diem to a new level of flexibility.  I'm also stunned by the other enhancements Thibault, euh Stéphane has added to the admin queries, a big thumbs up!
I'm currently thinking of enhancing all front-end widgets with a credential option like the pages form already has, for me a great feature as it enables us to control much more wich widgets are available in the frontend to registred users.
@Stéphane: how do you think about this? Pbbly best to make this configurable and disabled by default?
@Thomas: I'll get in touch with you soon about the DmForumPlugin

Keep up the amazing works, a big holla for Stéphane!

Erik

On Jan 10, 2011 12:31 PM, "Thomas Ohms" <thoma...@googlemail.com> wrote:

Stéphane

unread,
Jan 10, 2011, 9:00:40 AM1/10/11
to diem-...@googlegroups.com
Hello,

The changes are not yet stable, I see some things which are worth changing to make the code better.
In the mean time, writting some API doc would help you in helping me writting better code... And finally having a better Diem =)
So I'll stop for a week to write changes and help you get the stage about these new bits of code.

Is it all right ?

P.S.: I have commited some devs on 5.4, index action lists get ajaxified, and I added the pagination & searching of m-to-m in edit forms, a vid here :
These (I hope) means better UI for end-users and (I'm sure) less server mem+cpu+bandwidth consumptions :)

Regards,

Before Printing, Think about Your Environmental Responsibility!
Avant d'Imprimer, Pensez à Votre Responsabilitée Environnementale!


Reply all
Reply to author
Forward
0 new messages