MySQL Connector Plugin
Torsten Ziegler
news for the adventurous, the first version of my new plugin is available.
See: http://www.ziegi.de/deepamehta/ and
http://ziegi.de/deepamehta/deepamehta3-mysqlconnector.html
It works only with the old stable Deepamehta v0.4.4 maybe also v0.4.3
With a small database set it seems to work almost as I want it to do.
Using my production database (3 differnt topic types with a total of
approx 15000 entries, and approx 45000 relations)
I stumble upon some limits of the DeepaMehta 3 architecture.
Interactions with Topics that hold more than some hundreds
of Related Topics is very slow, even using chrome/chromium as browser.
One limiting element is browser/javascript speed, the other seems to be
the neo4j database backend. On reading/deleting
some 10 thousand Relations my harddisks make strange noises as if the
disk access would be very fragmented and in very
small chunks. I seems to be a problem especially with the Relations, as
creating some 10 thousand Topics seems to be no
problem. See also the ToDo and Known Bug sections in the above linked
documents.
I hope it is of some use for you,
Torsten
--
Torsten Ziegler
tor...@ziegi.de
Jörg Richter
Hi Torsten,
Hey, this is great!
Congratulations for your plugin!
For the moment let me help to solve one of the Known Bugs you describe at
http://ziegi.de/deepamehta/deepamehta3-mysqlconnector.html
> Automatic creation of custom Topic Types does not work yet
> This part of the code still bugs me:
>
> // create a copy of the standard import topic type with the user specified uri
> TopicType standard_import_type = dms.getTopicType("de/ziegi/mysqlconnector/topictype/mysqlconnector-topic", clientContext);
> TopicType new_topic_type = new TopicType(standard_import_type.toJSON());
> new_topic_type.setTypeUri(topic_type_uri);
> new_topic_type.setLabel(topic_type_uri);
> dms.createTopicType(new_topic_type.getProperties(), new_topic_type.getDataFields(), clientContext);
>
> The Topic Type gets created but the database breaks with the following error:
> Graph inconsistency for topic type ... there are 9 nodes in data field sequence but 6 meta property nodes
Indeed, copying an entire topic type definitions this way was not supported.
The problem were the plugins who add their specific data fields to newly created (copied) topic type. Like the Time plugin with its timestamp fields. Indeed, these fields were added twice and corrupt the topic type (as you suggested in your recent mail).
Now the DM core is fixed in this regard. Data fields are not added if added already. This is a quick workaround that works for the moment (as proved by the new copyTopicType() testcase in deepamehta3-example-plugin).
The downside: In order to benefit from this fix you must adapt your plugin to the master branch of the upcoming DM3 v0.4.5 release. There are some changes in the core service API (e.g. dedicated classes like Properties, PropValue, CommandParams, CommandResult replaces the generic Object and Map parameters). Adaption is straight-forward. The storage format has *not* changed, so you can keep your DM-data. Another good news: setting up a DM development instance and development itself is much easier than in v0.4.4, e.g. all your changes are hot-deplyed without interacting with Felix (and the manual Felix installation is no longer required). The new setup process for developers is described in the Wiki: https://github.com/jri/deepamehta3/wiki/DeepaMehta-3-Development
If you need any help with setting up and/or migrating your code to the master branch please post here.
BTW: copying an entire topic type through its JSON representation is little odd (JSON is meant solely for serialization when sending types over the wire). But I appreciate your creativity and I'm happy it works at all :-)
From my point of view copying a topic type is rather complex and requires a proper solution in the future. This could comprise of two parts:
a) extending the DM data model to differentiate between data fields *originally* defined by a topic type, and *auxiliary* data fields that are added dynamically by the installed plugins. Handling the auxiliary data fields should be exclusively in the responsibility of the respective plugins.
b) extending the DM core service by a copyTopicType() call, that copies only the original data fields (and let it to the installed plugins to add their specific data fields).
For the moment the DM data model makes no distinction between original data fields and auxiliary data fields (so your current approach is absolutely reasonable).
One thing to consider: your plugin is much more than a MySQL-connector. You might want to stress that by renaming it. MySQL-connector suggests it is just another connector for the DM storage (that is MySQL as an alternative to DM's Neo4j-based storage). But your plugin is rather about accessing and integrating external data sources that contain user data and that exists independantly from DM. As Malte suggested recently your plugin is thus a first incarnation of DM's "Datasource" concept. To reflect that it could be named e.g. "dm3-datasource-mysql". What do you think about that?
Cheers,
Jörg
x28
my WordPress notes a little easier again. With this Query string
select id, post_title AS name, post_content AS data FROM wp_posts;
I could bypass step 1-3 of this http://www.deepamehta.de/wiki/en/old08
old description. But while the old DM2 express import wizard (see
http://deepamehta.newthinking.net/wiki/en/Image:import.wmv ) allowed
me to reveal many imported topics at once (by clicking at the bottom
of the search results list), I don't know how to do this with the new
features. Am I missing something?
Matthias
Torsten Ziegler
thanks for your support,
and for implementing changes in v0.4.5 to make my plugin run smoother.
I am aware of some changes in the new version as I started my development
with the up to date code. But as I need the plugin for the stable
version, as I have
to use it on my production database I switched back to v0.4.4
As soon as a new stable version will be published I will update the
plugin to meet
the new core service API.
I might need some help with the migrations then as I don't really
understand them yet.
Two points on my wishlist for the next version are already mentioned in
my ToDo section:
- a way to exclude topic types from the create menu
- a way to register custom document renderers
both things are already prepared in your code but are still private
functions a plugin
has yet no access to.
If the copying of the topic type (you implemented the support for) is to
strange
I could also implement it by creating the new topic type from scratch.
Also about renaming I am quite open to your suggestions.
Cheers,
Torsten Ziegler
thanks for your feedback
Am 04.03.2011 22:26, schrieb x28:
> Thanks, Torsten, for the progress. I could finally make the import of
> my WordPress notes a little easier again. With this Query string
I am glad the plugin matches your use case
> select id, post_title AS name, post_content AS data FROM wp_posts;
you could even omit the AS clause and just return the three columns
> I could bypass step 1-3 of this http://www.deepamehta.de/wiki/en/old08
> old description. But while the old DM2 express import wizard (see
> http://deepamehta.newthinking.net/wiki/en/Image:import.wmv ) allowed
> me to reveal many imported topics at once (by clicking at the bottom
> of the search results list), I don't know how to do this with the new
> features. Am I missing something?
No, you are not missing something,
I did not know of your use case yet, so this is not implemented.
Though I did think of this feature but did drop it as reveiling
thousands of topics (in my case) would not be usefull.
I wonder if this plugin is the right place to add the functionality.
As I also work om some other extensions I could imagine a plugin
that adds a "reveil all directly related topics" button either to the
context menu or the detail pane of all topics.
By this it would be possible to chooses one of the "MySql import helper
topics"
and reveal all imported or changed topics. While it would also apply to
all other
topics and allow to unveil all topics of a search result for example.
One question that comes to my mind is how the placement should be then.
All known but hidden topics should be shown at their respective places
and all unknown topics should then be placed by the grid algorithm?
What do you think?
Cheers Torsten
--
Torsten Ziegler
tor...@ziegi.de
x28
Matthias
Jörg Richter
On Mar 5, 2011, at 9:58, Torsten Ziegler wrote:
> If the copying of the topic type (you implemented the support for) is to strange
> I could also implement it by creating the new topic type from scratch.
Yes, in case you don't need a generic approach for copying Topic Types (I talked yesterday about) you can create the new type from scratch by copying the source type's *original* fields one by one:
TopicType sourceType = dms.getTopicType("de/ziegi/myplugin/topictype/mytype", clientContext);
List dataFields = new ArrayList();
dataFields.add(sourceType.getDataField("de/ziegi/myplugin/property/myprop1"));
dataFields.add(sourceType.getDataField("de/ziegi/myplugin/property/myprop2"));
dataFields.add(sourceType.getDataField("de/ziegi/myplugin/property/myprop3"));
Map properties = new HashMap(sourceType.getProperties());
TopicType newTopicType = new TopicType(properties, null);
newTopicType.setTypeUri("de/ziegi/myplugin/topictype/newtype");
newTopicType.setLabel("New Topic Type");
dms.createTopicType(newTopicType.getProperties(), dataFields, clientContext)
Important here is to create a new properties set (new HashMap) in order not to override the (memory-cached) original's type properties.
This should run in current stable v0.4.4
Cheers,
Jörg
Jörg Richter
On Mar 5, 2011, at 9:58, Torsten Ziegler wrote:
> Two points on my wishlist for the next version are already mentioned in my ToDo section:
> - a way to exclude topic types from the create menu
> - a way to register custom document renderers
> both things are already prepared in your code but are still private functions a plugin
> has yet no access to.
That's great your plugin development shows where the client (resp. core) requires implementation changes! Your requests are welcome.
Torsten, what about commiting the required changes yourself? It could speed up things. You could fork the deepamehta3 project on github or I could configure you as a co-worker and you setup a branch in our repository. If you need help with git/github we're happy to support you.
In any case, you must migrate your plugin to the current development (master) branch. Release of 0.4.5 is imanent (this week or next) and the API is already fixed for this release.
> Also about renaming I am quite open to your suggestions.
Fine! How about dm3-datasource-mysql (Maven artifact name)?
Your plugin is more than a connector, and more than an importer, as it connects, imports *and* refreshes (!) contents of a DM-independant external datasource, MySQL in this case. So dm3-datasource-mysql.
Correspondingly your topic type "MySql Connector" could be named "MySQL Datasource".
What do you think?
Cheers,
Jörg
Torsten Ziegler
I have still more questions, one is concerning the migrations.
As far as I understand them they provide the interface for the
plugin developer to keep his topic_types up to date.
If a plugin gets installed the first time the
migration1.json in the resources/migrations directory gets loaded into dm3
but what happens on uninstall? As far as i see the topic_types
remain in the dm3 database.
What happens on a reinstall (vs. an update) of the plugin?
To change the topic_types of a plugin I will have to
adapt the plugin.properties file.
Could you please explain the "requiredPluginMigrationNr = x"
line.
Looking at the "Notes" plugin (dm3 v0.4.4) I can see the java code for
the migration.
Could you please explain the logic that has to be implemented there ?
As far as I see it there are only two states:
a) the topic_type is not known yet, so it gets created using the
migrations.sjon file
b) the topic exists and gets update without any further checks
What happen on a reinstall/update of this plugin,
will the code in state b) be reexecuted and render the topic_type with
doubled DataFields ?
How can I check which version of the plugin has been installed before
(and what in case
the plugin the plugin has been uninstalled in the meantime, but
topic_types have been created) ?
Quite a lot of questions,
my use case is the following:
- plugin XX in Version 0.1 implements a topic_type in the
migrations.json file
- the plugin gets installed and the user creates some topics
- plugin XX in Version 0.2 changes the topic_type and implements the
java code for this
- some users install the update, others deinstall the plugin (because
they don't like it)
- plugin XX in Version 0.3 changes the topic_type again and implements
the java code for this
- some users install the update (some of them running version 0.1 others
0.2),
others install the plugin again (after they deinstalled one of the
previous versions,
because they like the new features)
Some there has to be a distinction which version of the topic_type is in
the database.
Maybe I am blind but I can't see this yet.
Torsten Ziegler
one thing I missed in my last use case is:
- the user might change one of the topic_types a plugin
provided by using the Topic Type Editor
Should we try to limit access of the user or make the
plugin/migration scenario smart enough to handle this?
For instance I created a topic_type per migration with a
html-text data_field and changed it later on to a text-multiline
data_field to speed up my interaction while editing.
So maybe we need two more information in the topic_type
a) which plugin version did create it
b) did the user or plugin create or change it
But I guess this goes far beyond my initial question
concerning how to write a proper migration.
Jörg Richter
Hi Torsten,
thank you for your questions!
Let me provide some general information about migrations here.
ABOUT MIGRATIONS
Migrations are used to keep the DB's data model (topic types) in-sync with the software version. A migration represents a data model version and additionaly provides the instructions to update to that version. The software (every plugin as well as the DM core itself) knows on what data model version it relies on. It is the responsibility of the DM core to compare the actual version (stored in the DB) with the version required by the software (configured in the plugin) and to run all the migrations needed.
The user is not required to install every plugin update. Migrations are identified by consecutive numbers: 1, 2, 3, ... If e.g. plugin version 3 (= migration number) is currently installed in the DB and then updated with plugin software version 7 the DM core will run migrations 4, 5, 6, 7 in sequence. This is handled automatically.
WRITING MIGRATIONS
You, the plugin developer is responsible for 2 things:
1) Tell DM what migration number is required by your plugin. To do so set the requiredPluginMigrationNr in resources/plugin.properties to the very number. It's that very number that activates the DM core's migration machinery.
2) Provide the migrations itself. There are two kinds of migrations:
a) Declarative migrations. These are handy if you just want introduce new topic types (as well as topic instances). In this case you need no Java code at all, but declare the topic types (and topic instances) in a JSON file resources/migrations/migration<nr>.json
See the deepamehta3-accesscontrol plugin for an example.
The data format of the JSON file is documented in the Wiki:
https://github.com/jri/deepamehta3/wiki/Data-Format
b) Imperative migrations. Sometimes your migrations manipulate existing topic types (e.g. by adding/removing data fields) or involve further DB examination and/or logic. That's when imperative migrations come in. An imperative migration is a Java class that adheres to some pattern:
- The class is public and its name is Migration<nr>
- The class is contained in package "migrations" within your plugin's package
- The class extends de.deepamehta.core.service.Migration
- The class contains a public void run() method
Within the run method you have access to the DM core service through the "dms" object.
See the deepamehta3-workspaces plugin for an example.
Caveat: if your plugin provide imperative migrations but does *not* provide a Plugin sublass you must configure the "pluginPackage" property in plugin.properties to the package that contains your "migrations" package. Otherwise the DM core can't find your migration classes.
See the deepamehta3-notes plugin for an example.
CONFIGURE THE MIGRATION RUN MODE
Sometimes it is necessary to run a migration only when the plugin is *updated* but not while a "clean install" or vice-versa. This can be configured for a single migration in the resources/migrations/migration<nr>.properties file. Only one property is expected here:
migrationRunMode = CLEAN_INSTALL | UPDATE | ALWAYS
Default is ALWAYS.
For an UPDATE example see the "contacts", "topicmaps", and "workspaces" plugins along the comment in the respective migration class. For a CLEAN_INSTALL example see the "iconpicker" plugin and the comment in the Migration2 class.
FACTS AND HINTS TO SUPPORT MIGRATION UNDERSTANDING:
There is no one-to-one relationship between plugin release and migration number. A plugin release can provide any number of new migrations (a mix of declarative and imperative ones).
How the DB tracks migration numbers of installed plugins? Answer: internally for every installed plugin the DM core creates a topic of type "de/deepamehta/core/topictype/Plugin" and stores the migration number in its "de/deepamehta/core/property/PluginMigrationNr" property. When a migration runs successful the DM core increases the migration number accordingly. For the plugin developer this is considered opaque.
When the plugin representing topic is created in the DB the migration number is initialized with 0.
Versions (= migration numbers) are tracked on a by-plugin basis, *not* by-topic type. A plugin knows what topic types belong to it and what their (model) states are at any migration number.
The DM core ensures every migration runs only once. The migration numbers are ascending strictly. They are never decreased.
Plugin deinstallation: for the moment the DM core provides no deinstallation procedure for plugins. This is a future subject.
If your plugin don't need migrations the requiredPluginMigrationNr property must not be configured. The required migration Nr is assumed to be 0, thus no migrations will be run.
Plugins must prevent the user from interactive manipulations of topic types that are not supported by the plugin's application logic (possibly by the means of the deepamehta3-accesscontrol plugin).
Develop with migrations:
a) Keep in mind a migration number is an assurance of a DB (model) state. If you change a migration that has already run, clear your development DB to enforce running it again.
b) To enable arbitrary plugin updates (version skips) keep track what plugin release required what migration.
Cheers
Jörg
Torsten Ziegler
thanks for this great in depth explanation of
the migration process.
This is a perfect clarification for us plugin
developers.
Thanks a lot !
Torsten
> J�rg
>
--
Torsten Ziegler
tor...@ziegi.de