Developing UCM components & duplicate data?

[reynaldo celaya III]

I understand UCM is still in the works, I might be early but im interested in developing/updating some components for joomla 3.1 that use the ucm_content table
Copying what i saw in com_content I have implemented the tags feature (which was very simple) into my components/experiments. Upon saving tagged items the abstract content table #__ucm_content now contains duplicate content from my components table(just like after saving a tag in com_content).

Im wondering why the #__content (title, alias,etc) feilds are duplicated in ucm_content, when #__content should only contain the fields that are unique to com_content.
Is it inteded that there be duplicate data or is it for compatibility(so we dont break com_content)?

I envision ucm being a common table that developers use as a base to handle the common data(publishing, state,check in/out... etc) and only have to define data tables that contain fields that make their model unique. When querying we simply join the tables..... No more modeling closely identical tables to support common data only what makes our data unique.
I want to use things as intended and not go anti-pattern, so is it safe to use the ucm tables as described above?
Is joomla planning on using the ucm in this way?

Thanks in advance to anyone that wants to discuss using the ucm
RC3

[Mark Dexter]

Hi. As you note, we are at the beginning of a transition with respect
to UCM. After a lot of discussion, it was decided for version 3.1 to
just duplicate the data for tagged rows in the ucm tables. The plan is
that in a future version, (not sure which one at this point) we will
just drop the old #__content, #__weblinks, #__newsfeeds, and
#__weblinks tables, perhaps replacing them with database views the
pull the old column names from the new ucm-based tables. In the
meantime, there will be duplicated rows, but only if you use tags and
only for tagged items.

If you are doing a custom component, you could just use the ucm
tables. The plan is that the ucm tables will hold all of the common
columns and that if a component requires non-standard columns, these
would be stored in a custom table for that component. Since we haven't
implemented that yet in core, I don't know if we have any naming
conventions or other patterns determined at this point. I think as
long as you follow this general pattern, it will be fine.

Hope that helps. If others have more information, please feel free to
jump in. Thanks.

Mark

> --
> You received this message because you are subscribed to the Google Groups
> "Joomla! General Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to joomla-dev-gene...@googlegroups.com.
> To post to this group, send an email to joomla-de...@googlegroups.com.
> Visit this group at
> http://groups.google.com/group/joomla-dev-general?hl=en-GB.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[elin]

Yes the intent is to eliminate the duplication (and perhaps also the duplication in smart search). And yes the goal is to solve a lot of problems including duplication of task and lack of a unique id. 

The intent is that eventually in the core we will drop the use of the old tables for anything but "special" fields. And amazingly even now you can migrate the data to that table just by tagging everything so for once we actually have a basic migration strategy in place.  

However for people with websites there are tons of modules and plugins using the data in those tables as overrides and other work that they have done, so we obviously must move very carefully and slowly along the way of accomplishing movement to a ucm table for the core components.   

Thus besides the front end tags layouts and module the core is not using this data yet.

  Developers can start using ucm_content by itself right now if they wish, and one of my hopes is that we will see some use of that (and JTableCorecontent and JHelperContent and the JUcm package) pretty quickly.  The great advantage of people doing that is that it tells us what developers need and we may even get code contributions to facilitate that. I see no reason in the world not to use the code and table as you see fit, as long as we all know that there will certainly be a new set of apis added and many small decisions made along the way.

As Mark pointed out (his post came through as I was writing) some of the complexity of the current work is that it is not copying everything, just tagged items, which is a commitment we made to people concerned about storage. 

Elin

[Amy Stephen]

Not so much a concern of storage as it is of data integrity. Which data is official?

[Mark Dexter]

Right now the ucm data is only used for displaying the tags layouts.
After the next phase, the ucm data will be official and the old tables
will only be for b/c. Mark

[reynaldo celaya III]

Ok, sounds like we are on the same page. For the longest time i have used standardized field names to achieve what this makes possible.... features for free.

Thanks for the answers! My next model will use the ucm table and I will post any issues I come across. back to work

[elin]

If you want to make sure you don't miss anything in implementing tags in addition to looking at content you could look here (corrections, additions very welcome) http://docs.joomla.org/Using_Tags_in_an_Extension

Elin

[Amy Stephen]

Thanks, Mark. Quick look at your implementation -- good directions.

[Ove]

I'm in the same situation as Reynaldo but one step further. I've migrated testdata to the ucm tables.#__ucm_base and #_ucm_content. (Manually but I think (hope) I got them right) I have a third table with "special" data with a 1-1 relation and with the same key-id. I created two entries in the #__conent_type. One for the categories and one for my table(view)

1: do I have to set any field mappings in the type table and what is here the special? My guess is that it has nothing to do with my "special" table with extended data. I do have to handle this table by myself as before?

2. There are a lot of id's involved in the #__ucm_content. The one I'm not sure if I have to fill is the core_content_item_id. My guess is that it's the id from. e.g. #__content and I do not have to fill anything in as those data are in the Ucm only.

3: I think there are only two other fields I didn't yet use in my old "content"-table. I just added a future use of featured but do not know any use of xreference - kept for historical reasons? Is it used somewhere in com_content? I found nothing.

Thanks to anyone who can give a hint.

Ove

[Donald Gilbert]

It's discussions like this that really concern me about UCM and how the developers are using it in the community. I'm all for getting it ready for them to use because it's a great feature, but there's not a clear API right now.  Developers really have to be careful to be sure that they're only manipulating their own content that they add to core_content.  you know there is potential to accidentaly wipe out users content through misuse of the existing API if we're not careful. So I think the next step for UCM really should be building that rock solid API that is extremely easy to implement.  

—
Sent from Mailbox for iPhone

[Mark Dexter]

I think that's a great idea. Maybe we can discuss what that API should
look like? Version 3.1 just has what was needed for tags, I believe. I
know that Elin and David and Matias (and maybe others) have had some
discussions about this, but I don't know if there is anything written
up at this point. But now is the time to get this API sorted out, for
sure. We have a Unified Content Production Working Group on paper at
least (http://docs.joomla.org/Production_Working_Groups#Unified_Content_Model_Working_Group).
This sounds to me like a good next step for this group.

Mark

[Ove]

I'm perfectly aware of the state of Ucm for developers in v 3.1 ---
Testing only --- But to do that it would be nice to have that information.

I understod tests as welcome. And quote from the mentioned pages:
"This is a working group for looking at how the CMS could take advantage
of UCM. "
"Work on the CMS 3 series to implement UCM for the core and extensions
that wish to opt into it. "
"Work on a the CMS 4 series that would be built around this model as
well as other architectural changes. "

Best regards
Ove

Mark Dexter wrote On 04.05.2013 19:16:
> I think that's a great idea. Maybe we can discuss what that API should
> look like? Version 3.1 just has what was needed for tags, I believe. I
> know that Elin and David and Matias (and maybe others) have had some
> discussions about this, but I don't know if there is anything written
> up at this point. But now is the time to get this API sorted out, for
> sure. We have a Unified Content Production Working Group on paper at
> least (http://docs.joomla.org/Production_Working_Groups#Unified_Content_Model_Working_Group).
> This sounds to me like a good next step for this group.
>
> Mark
>
> On Sat, May 4, 2013 at 10:07 AM, Donald Gilbert <dilber...@gmail.com> wrote:
>> It's discussions like this that really concern me about UCM and how the
>> developers are using it in the community. I'm all for getting it ready for
>> them to use because it's a great feature, but there's not a clear API right
>> now. Developers really have to be careful to be sure that they're only
>> manipulating their own content that they add to core_content. you know
>> there is potential to accidentaly wipe out users content through misuse of
>> the existing API if we're not careful. So I think the next step for UCM
>> really should be building that rock solid API that is extremely easy to
>> implement.

>> �

[Bakual]

I'm not at my computer atm but I can answer the field mapping part. It's used by the tags component to create the ucm entries based on an existing table. Since fieldnames don't necessary match, you have to map them. Special relates to the fields not available in the ucm table.
Hope this helps

[Ove]

Hi Bakual,

I got that. But I want to store 'common' data in the Ucm table only and
the rest in a separate table. (I used two tables for the item already,
where one was almost identical with the Ucm-table.)

I've got the item-list and the edit views working including tags. Next
step - >store the data with and without tags. I'll see what happens.

I guess it would be possible for a (future) Ucm-Api also to take care of
the "special" table. With a Ucm-form-Xml and adding a "special" Xml it
should be possible for an Ucm-Api to keep developers away from saving to
the Ucm-tables with own methods. Let's see what's coming.

Ove

[elin]

JTableCorecontent should handle save fine by itself without the mapping--the common mapping is really solely for translation/migration purposes i.e. if you are not using the store() from a different table.  Special is solely for non common fields and will become more   and will but right now since we are not doing that in the core.

If you are using the UCM table only you don't need "special" and you can just leave it with an empty array as it is now. As noted in the docs the intent is to change to array or object with a default to object (There is code in the tracker for this that may even have been merged already). You also should not need common but I'll explain below why right at this moment you do. 

Right now in the core we are initiating store() in the other table and saving in ucm_content, so you are pushing ahead with where we want to go which is initiating the store() in ucm_content/JTableCorecontent and then only storing special if it applies.   In that case you should not need mapping at all since any differences internal to the individual fields should be managed by the extension or type (once we have a ucm extension proper it will support different types).  That is for example of course there are differing structures of params, images, uls, and (I expect) body to name a few. 

Right now using the tags api requires mapping because it is really designed for 3.0/3.1 extensions, but it should of course work with storage only in the shared table without mapping as part of really moving to using that table.  

If people want to be in on actually working on ucm implementation please make sure to send me your email.

Elin

>>> email to joomla-dev-general+unsub...@googlegroups.com.

>>> To post to this group, send an email to
>>> joomla-de...@googlegroups.com.
>>> Visit this group at
>>> http://groups.google.com/group/joomla-dev-general?hl=en-GB.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>>
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Joomla! General Development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to joomla-dev-general+unsub...@googlegroups.com.

[elin]

Actually I just want to mention something about accidents of history. As many people know I was very focused on UCM for perhaps 18 months, but then when Mark  asked me to work on tags I put it aside and really worked on nothing but tags for many months (hence little progress on UCM). If you looked at the first implementation of tags it did not use UCM at all but implemented a different workaround of the many tables model. However at a certain point the people working on tags said, you know it makes no sense to work around this issue once again, we should just make a common table.  As we all know, literally making a shared table is not the hard part (smart search already does this and solves the different column naming problem in a different way and of course I had migrated all of the sample data in some of my example ucm applications) , but it is a necessary first step. So that is really all that has happened, and now we have an opportunity to work out from a database table that has data and a method of migration to actually building an API for ucm.

Another important preliminary set of task for implementing UCM is to identify and eliminate duplicate code via abstracting to higher levels whether that be to our cms library, the legacy library or JLayouts.  By really identifying what is really different about extensions and what is the same we can build an API  that is really effective and useful for everyone.   Hence you see me and others putting in a lot of PRs for that work (and I strongly encourage other people to help with that).   I'm pretty much looking at everything and asking "why don't all of the core types have that?" or "is that really necessary"?  For example, why don't all core extensions have featuring? It seems to me that it's useful for curating any kind of content and it would be great to manage across types if one wished. On the other hand the weblinks go() method (really from what I can tell the only unique thing about weblinks) .. is that useful generally? Is it needed still?    Is there a reason for a different ordering field in each component? So I encourage people to look pretty carefully at the core extensions and their own extensions and ask these same questions.

Now that tags is relatively settled I'm starting thinking about UCM again and I'd love people who want to be on a WG mailing list (however for working not for lurking) to be involved in thrashing out ideas about this.

Elin

[Amy Stephen]

I understand the comment "working not for lurking" -- but it's also good to remember that devs learn and can ask questions, point out issues when following along.

I think it's perfectly fine to close down discussions for a working group since it can get pretty noisy and you can spend a lot of time answering questions instead of coding but the flip side to this is people are blind-sided during a release.

Maybe a happy medium is to code in the open, make it clear where the code is, work there, and then handle questions on the normal list?

If nothing else, people can't complain if the code is available during development.

--
You received this message because you are subscribed to the Google Groups "Joomla! General Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-gene...@googlegroups.com.

[Ove]

Thanks for the info Elin,

I'll now have to debug to find the right methods for saving to the Ucm ,
Tags and my "special" table. I think I have'em in my head but have to
check it deeper. My model is growing and growing and .... I'll try to
find a way to shrink it by reusing more method code.

I'll contact you. Maybe I can contribute in some way.

Best,
Ove

[reynaldo celaya III]

@elin Id like to join your WG, let me know when a branch/fork has been established.

@over, are you managing saving the two tables(#__core_content,#__your_extension_special_data) in your model. Do you use JUcmContent?
Whats your thought process with handling the save?

I see that the mappings inside #__content_types should enable us to pass data and content_type to a single object that takes care of either (writing the insert/update sql) or (instantiating the two(possibly more) table objects and managing the save ). Your thoughts?

Do you think this should be done by building out JUcmContent to manage the saving/retreiving data, or by a new UCM helper Class? It seems that right now as elin said JUcmContent only handles the migration(duplicate data), not necessarily saving/mapping extended special data(correct me if this is wrong). In my tests with my current content type I have only one common field mapped : "common":[{"core_content_item_id":"id"}] and my special field is a FK that references another #__ucm_content id: "special": [{"ref_item_id":"ref_item_id"}, Nothing is saved in my extended table when I passed data to a JUcmContent instance save() method.

What are the intents of the #__ucm_base, Backwards compatibility, easy language indexing,many-many relationships?? There is already the 1-1 core_content_item_id that can be used to join the tables?

I think we should start with use cases, such as self referencing FK's, or multiple primary keys, no primary Id, many-many relationships etc.... and work back to basics(on-to-one extended data)

Lets get this going, the potential of this is great. I think of never again writing the bulk of administrator mvc and only what makes our components unique(we can make this better than django(python orm) admin). Also heirchical layout/templating potential using hmvc.

Thanks again everyone. @elin let me know when everything is set up.

[Ove]

Yes I have it working as a "calf on green manures". Or what do you say in english?

An advice to developers not wanting to be early birds and test it to get a first idea of the use of an Ucm ( well I wanted to ), wait until com_content is using the Ucm tables as main tables. BUT! start implementing the tags!

I think Ucm includes great opportunities for developers disliking coding, like myself. Including out-of-the-box extensions to your component. Wouldn't it be great? An obvious example would be a comment extension to Ucm-content.

First: What I write is NOT intended as any criticism of anyone person or group. Missing or do not work stands only for my own thoughts at this VERY early start of Ucm.

Well, I do not think we should set the short term goals too high. I really never understood the single table only approach in Joomla so to involve a couple of tables is a good start.

In short here my experiances when trying to adopt a components main data to the Ucm tables (only!).  As I wrote before there were no issues on the data since I had adopted the fields from com_content long ago and they were devided in two tables Main and Details, 1-1 relationship. You know all those dates, access, state, user-ids aso were there:

I asked before about all those keys and id's in the tables. It turned out to be The problem.  As far as I know a table key was always 'id' in all tables, now you have to handle a couple of them. Yes! The Joomla libraries handles different table keynames but I think most of us got very used to an 'id'. e.g. you have to handle the requests differently. I know one example in existing code where an a_id or an id is handled as value for the same key. Why not unique aliases in the Ucm table? I found one break in the Reorder methods because of the keyname. I also earlier got the impression that a unique key ( = id ) is needed in all tables. My impression! The both Ucm tables have the same key value. Good, I also use this number in my "special" table. No other reference. It might be that the old key has to be kept for transition reasons. Another problem with this is that if a record is new or not is handled by the fact that the key has a value or not. You get a isNew back as a state but still have to use some tricks. The Ucm stores the ucm_content with the Api and then "hardcoded" the ucm_base. After saving the tables I have to update a couple of tables because at least one needs the key value. For now I'm looking for a problem with my tags but this should be a minor one, related to my form/metadata.


@reynaldo celaya III as you are asking

 are you managing saving the two tables(#__core_content,#__your_extension_special_data) in your model. Do you use JUcmContent?
Whats your thought process with handling the save?

To  be able to follow the Tags description as tightly as possible and use the Api as is,  I have an own table class MycompTableMycontent extends JTableCorecontent
with store and delete. Using this is the first step in my model. Now I have my data in Ucm and in Tags and a key-value and a isNew back from saving. Next I save my Specials using a SaveMySpecials method that takes a isNew parameter.  Instantiating the MycompTableMyspecial extends JTable. This so far has only the method.- StoreDetails( $newItem, $updateNulls = false) that updates or inserts the record. We are not ready yet! ;) As last step we have to handle the multilingual associations. Ready!!

If those steps would be handled by the Api with only a few lines of code it would be great! You see, It is a very normal save process but a lot of coding for each and every developer.

I see that the mappings inside #__content_types should enable us to pass data and content_type to a single object that takes care of either (writing the insert/update sql) or (instantiating the two(possibly more) table objects and managing the save ). Your thoughts?

You shouldn't need any mapping. You define the field names in the form and the api bind method handles filling the values in..

Do you think this should be done by building out JUcmContent to manage the saving/retreiving data, or by a new UCM helper Class? It seems that right now as elin said JUcmContent only handles the migration(duplicate data), not necessarily saving/mapping extended special data(correct me if this is wrong). In my tests with my current content type I have only one common field mapped : "common":[{"core_content_item_id":"id"}] and my special field is a FK that references another #__ucm_content id: "special": [{"ref_item_id":"ref_item_id"}, Nothing is saved in my extended table when I passed data to a JUcmContent instance save() method.

Saving I think is answered above. Retrieving data could be standardised. The only question I have here is the size of the Db return. There has to be some kind of field reference to not get GB large data returns on lists. 

What are the intents of the #__ucm_base, Backwards compatibility, easy language indexing,many-many relationships?? There is already the 1-1 core_content_item_id that can be used to join the tables?

The inventor has to answer that. So far I have not seen the use of it. Tags?

[reynaldo celaya III]

 @ove, you seem like a good person, thanks for taking the time to reply. Did you get your component working?

I think Ucm includes great opportunities for developers disliking coding, like myself. Including out-of-the-box extensions to your component. Wouldn't it be great? An obvious example would be a comment extension to Ucm-content.

 

I would rephrase that, "for developers that dislike coding the same features over and over again to support tasks like check-in/out, state changes, access, ownership, version, featured, etc". I love coding, and think others will love coding with this framework when they get to focus on the innovative aspects of their extensions.
 

Well, I do not think we should set the short term goals too high. I really never understood the single table only approach in Joomla so to involve a couple of tables is a good start.

Agreed, there are many models that dont fit in with ucm, either requiring simplicity or complexity. But within the CMS there are alot of occasions where it doesn't make sense not to use it. But i disagree with setting goals too high, this is brand new...the decisions that happen now affect the flexability and maintainability of joomla as a whole in the future. Look at JTable and its rigid dependency on a single primary key (not a deal breaker).
 

In short here my experiances when trying to adopt a components main data to the Ucm tables (only!).  As I wrote before there were no issues on the data since I had adopted the fields from com_content long ago and they were devided in two tables Main and Details, 1-1 relationship. You know all those dates, access, state, user-ids aso were there:

same here, many of my previous tables are very close to #__content

  For now I'm looking for a problem with my tags but this should be a minor one, related to my form/metadata.

Yea, right now it seems there is a dependancy of having your form xml structured with a <fields name="metadata"> with your  <field name="tags" type="tag" ..... ></field> inside of it.
       

You shouldn't need any mapping. You define the field names in the form and the api bind method handles filling the values in..

 
in the way you are handling the two tables this is true, in your situation using JUcmContent mapData() woud make the bind slightly more efficient(only passing the data that can be bound). The save can be managed by a single object by using the metadata in #__content_types.table field to instantiate the tables, then use field_mappings to efficiently bind the data to each respective table save the extended data and then save the core data.

NOTE: mapData() never maps special data.... there is an unreachable code block. I submitted Tracker item
http://joomlacode.org/gf/project/joomla/tracker/?action=TrackerItemEdit&tracker_item_id=30883

...
Saving I think is answered above. Retrieving data could be standardised. ......

Both can be standardized. Metadata about the underlying data model(relationships) is needed to create code that can handle multiple aspects, using the context created by the request(hope that makes sense). For the most part we only request of our models Display Item, Display List, Edit List, Edit Item, Display all, Display Categories, Display Category (Saving Item, sometimes handled in the controller?). Any variation of those are usually done through filtering/pagination. Every single one of these requests can be standardized in a flexible way by modeling the thought process that we make every time we code, deciding to return raw data to be bound to form elements or joined displayable data to be displayed in a layout or through JHTML objects. This idea is oversimplified of course but to get an idea look at other CMS in other languages, and solutions such as ORM. look at django models.

This will turn into a major architectural endeavor, so I love that other people are already diggin into it. This probably gets out of general development and more into CMS so i await the working group on UCM

[elin]

Great! Yes it is really more of a CMS discussion but it's important for people to think about what is useful for extensions.

If you look at how tags works you will see the potential for standardization of retrieving data. I think many may developers have kept close to #__content in structure and so it is quite easy to come into the ucm_content table.  

We have to both have big vision and remember that it is a step by step, careful, process that as long as we are in the 3 series is backward compatible and that provides clean opt in migration for developers who use the Joomla APIs. As most of you know we have worked very hard throughout 1.6 -2.5 to standardize the fields in the core content components, for example, because that is forward looking toward a ucm model.  There are many many small steps like that; as I mentioned moving to reusable layouts is another element of that. Anyway, there are quite a number of pieces involved if you start thinking about it, which the WG can discuss and take up.

I think it is so great how many people are diving in.  Your experience is going to be invaluable.

Reducing duplicate code is one of the great benefits of moving to ucm but we can also do a lot of that right now.

 Elin

[Ove]

As Elin wrote "Reducing duplicate code is one of the great benefits of

moving to ucm but we can also do a lot of that right now."

Is there a WG for multi lingual or is it more or less integrated with
Ucm? When picking in my model I found the code for item associations
(around 50 lines) that should be moved to the Cms-library including most
of other code for ML now completely within the component. As I remeber
there was a lot of copy/paste to get the associations inte the
component. Not much of adaptation.

To reynaldo celaya III and others with interest.
This is "NOT" about implementing tags in a component. You should really
adopt it as soon as possible.

I think I'll stop with the tests of using the Ucm tables here. By the
hands-on I've got the impression I wanted to.

I have my item-list and edit view working in backend. With tags. I made
my test-enviroment multi lingual but to get that working I saw that I
had to check most of the ML code if something breaks because of the
keyname or fieldnames. I started to but as it started to break I found
it not worth the time needed at this point.

Short conclusion. Very promissing BUT. If the fieldnames are not changed
in the next transitions it's a no go for existing components. There has
to be very, very good reasons for not removing core_ and not to use
'id' as main keyname. As I wrote before the key-name is a problem but
also every other fieldname just because of the core_ as prefix. e.g. I
had to change metadata to core_metadata. Then I didn't find a
work-around so I had to forge the tags.php to use core_metadata in
there. You have to change most fields in forms, you have to change field
names in 90% of all db Queries and last but not least you have to change
all output layouts. I don't think we have to argue about it? The core
components would also have to adopt the fieldnames. :) I guess some kind
of mapping layer wouldn't be a solution.

Everyone with interest please join to push the new iteraration(s) I hope
I can contribute with my small part.