Mesh4j use cases

[amsmota]

Hello.

I'm having a hard time trying to evaluate Mesh4j for our purposes,
allow me to do a quick overview:

We have to synchronize pieces of data between 2 systems, one based in
Java (my side) and the other on MS technologies. Our first thought was
to wrap our XML data in a SyncML message, send it to a MQ and each
side will handle the updates/synchronization on their one.

Since it seems to me that SyncML is a overkill for this, I'll like to
give FeedSync a try, since it has (I suppose) good support from
Microsoft and it has Mesh4j on my side...

However, I'm completely lost after looking at the documentation and
the examples available. I don't understand how I can "publish" the
changes in our data to a center location so that the other part can
read then, and the opposite on my side, getting from that location and
sync on my side. Should I drop the MQ and use a Mesh4x FeedSync server
to do this? Or actually 2 Mesh4x FeedSync server, one on each side?
Should I keep the MQ and write a adapter for it?

Any help will be greatly appreciated, specially pointers to code or
how-to's.

Cheers.

[amsmota]

Hello again.

I'm a little bit concerned about not having any feedback, I understand
that are only 29 people on this list, but nevertheless I was expecting
some feedback from the developers...

Anyway, I'm trying to put up a demo until the end of this week
consisting of a endpoint based on a MQ, for which I'll try to write a
adaptor, and another one based on a HibernateAdaptor, for which I'm
trying to understand how it works.

I already have a lot of specific problems, so I'll appreciate any
help. Not that I can't do it all by myself, but probably not until the
end of this week, and that's the deadline for a decision...

Cheers.

[Juan Marcelo Tondato]

Hello amsmota,

    sorry for the delay. I can help you. (We are working in documentation, "comming soon").

   

Mesh4x is based on feedsync specification, all the information is expressed as an Item (feed item).

 

An item has business data and sync information.

 

Item = content + sync

 

The content is a xml representation of your business data, currently we are working with rdf xml based on rdf schemas, rdf extraction is automatically generated by the adapters.

 

For sync information, Mesh4x works with a simple idea, you create a SyncEngine (feed sync protocol implementation) with a ISyncAdapter instance as a source and a ISyncAdapter instance as target.

 

ISyncAdapter is a CRUD Item interface (add/update/delete/get). For example your new MQSyncAdapter or SplitAdapter (with hibernate).

 

When you call to synchronize method the feed sync algorith is runned, as a result obtains a list of items with conflicts (items with versioning problems), in the best scenario this list is empty (no problems).

 

How to create an adapter to synchronize DB tables usign Hibernate and RDF schema:

 

   Use HibernateSyncAdapterFactory, call to createHibernateAdapter method.

   Parameters:

           String connectionURL
           String user

           String password

           Class driverClass

           Class dialectClass

           String tableName           ==> business table to sync

           String syncTableName   ==> table with sync information (automatically created if absent)

           String rdfBaseURL         ==> valid url, is used to create rdf schema

           String baseDirectory      ==> disk directory to save hibernate mappings, mappings are automatically created

  

  We are using a special hibernate feature to read and write db rows as xml (no pojos).

 

  The factory only create an SplitAdapter using HibernateContentAdapter and HibernateSyncRepository, mappings, rdf, and all is AUTOMATICALLY generated.

 

if you decide to create you own hibernate adapter or configure hibernate in other way, please see this method as example "createHibernateAdapter".

 

Hibernate adapters can be used without rdf, in this case the Item content is a plain xml. If you decide this strategy I can send you an example to create Adapter.

 

How to create an adapter to synchronize MQ:

    Create a class thats implements ISyncAdapter interface (you can use the abstract class named AbstractSyncAdapter).

    If you are using RDF, you need to transform the item content (rdf instance xml) to/from MQ objects.

    RDFSchema class has methods to do it (see by example HibernateToRDFMapping).

 

I hope to help you.

If you have doubts or problems please contacte me.

 

thanks in advance

JMT

 

[amsmota]

Many thanks for your answer. Let me tell you about the problems I had
so far:

1- this "sync module" I'm trying to build will work within a existing
app, so we already have a Hibernate SessionFactory available, so first
I tried to use it. I extended HibernateSessionFactoryBuilder for that,
but in several places of Mesh4j code there is a
Configuration.buildSessionFactory(), and since I cannot get a
Configuration from a existing SessionFactory I had to give up on that,
maybe later I pick it up again.

2- then I tried, as you point, to use
HibernateSyncAdapterFactory.createHibernateAdapter(). Several problems
here:

2.1- my table has lot's of relations that aren't loaded automatically.
I overcome this by putting all the hbm.xml files in the baseDirectory
and loading all

public static PersistentClass createMappings
(...)
File dir = new File(baseDirectory);
File[] files = dir.listFiles();
FileFilter fileFilter = new FileFilter() {
public boolean accept(File file) {
return file.getName().endsWith(".hbm.xml");
}
};
files = dir.listFiles(fileFilter);
for (File file : files) {
builder.addMapping(file);
}
(...)

2.2- even than, I allways get null on this line

mapping = cfg.getClassMapping(tableName);
if(contentMapping == null){ // TODO (JMT) create tables
automatically if absent from RDF schema
Guard.throwsException("INVALID_TABLE_NAME");
}

and I don't understand if the method createMappings should return the
"table" or the "table_sync". If it's "table", I get a NPE later on, if
it's "table_sync" it goes further away until problem 2.3. BTW, in the
*if* shoul be "contentMapping " or "mapping"?

2.3- i get a NPE in
public String getIdentifierPropertyName(String entityName) {
ClassMetadata classMetadata = this.getClassMetadata(entityName);
return classMetadata.getIdentifierPropertyName();
}
because at that point the entityName is the name of the table - say
"employees" - and the key in the sessionFactory.getAllClassMetadata()
map is the name of the entity - something like
"com.mycompany.blah.Employees".

So it seems that you're expecting that both tables and entities have
the same name, wich is not my case.

I'm going to hack the code a little more to pass this problem, but for
now I reaaly like to know what mapping, table or sync, should be
returned as per 2.2.

I'll try to follow the other examples you gave also.

Again, many thanks for your time.

[Juan Marcelo Tondato]

Remember that we are using a special feature of hibernate to read/write db rows as plain xml, is not neccesary to create pojo classes. See http://docs.jboss.org/hibernate/stable/core/reference/en/html_single/#xml.
 
This hibernate feature requires a differente hibernate mapping, this xml mapping is not the same mapping that you are using in a common java application when you use hibernate as ORM between classes and db tables.
HibernateContentAdapter and HibernateSyncRepository are based on this feature and is not easy to reuse in pojo scenario. All ours scenarios are based in tools that directly works with db tables and not mesh4x embedded in a business application.

We need to implements a new HibernbatePojoContentAdapter and HibernatePojoSyncRepository (similar to the existing one), this new split adapter components use a POJO SessionFactory (your bussines session factory), the core methods  read/write pojo objects. For create items we need to express the business objects as xml. RDF is added only if you really need it. The idea is create a SplitAdapter instance with a HibernbatePojoContentAdapter and HibernatePojoSyncRepository instances (sharing the same session factory).

Mesh4x is a open source that give help in humanitarian task, is a good adapter to collaborate with mesh4x.

If you need help please call me,

JMT

[Juan Marcelo Tondato]

I detect the following error in 2.2, 

 

 contentMapping = cfg.getClassMapping(tableName);

  if(contentMapping == null){  // TODO (JMT) create tables automatically if absent from RDF schema
        Guard.throwsException("INVALID_TABLE_NAME")

  }

 

thanks in advance

JMT

[amsmota]

I don't understand, is that the correct code? Because contentMapping
is declared as File and getClassMapping return a PersistentClass.

Cheers.

[amsmota]

Ok, so it seems that this won't apply to my use-case right now.
However, since at this time I only need a demo, I think I can go along
with the "table only, not pojo" architecture.

But even then I have errors. Having now setup correctly a
HibernateSyncAdapter, I'm trying to do a

ISyncAdapter repository;
Item item = repository.get("1000337000");

where "1000337000" is the key of a employees table, but I always get a
null. I do see a select on the _sync table that returns nothing.

So I think I'm missing something here. How the employees_sync table
know that something has changed in the employees table?

I think I'm not understanding the all picture here... I'm following
other posts with doubts I have in other situations also.

Cheers.

On May 27, 5:58 pm, Juan Marcelo Tondato <jtond...@gmail.com> wrote:
> Remember that we are using a special feature of hibernate to read/write db

> rows as plain xml, is not neccesary to create pojo classes. Seehttp://docs.jboss.org/hibernate/stable/core/reference/en/html_single/....

[amsmota]

So after playing a little more, it seems that all the applicable
scenarios are based on *automatic* replication between two endpoints,
like, say two tables in two databases.

However I don't want that to happen automatically, I want to trigger
it myself. So, I have a application with a Pojo, let's say that it
corresponds to only a table, and what I want is:

a) as a publisher:

If some-business-rule-is-true
*publish* this pojo/tabe row in a FeedSync format to a MessageQue.

b) as a consumer

When a new FeedSync Message comes in in a MQ i'm lisyening
consume the message
transform it into a XML/Pojo format
probably check some rules
create/update/delete this Pojo/table row

Is that feasible with Mesh4x?

I forgot to say in my previous posts that I'll be very happy if I can
contribute in same way to the project, specifically with the code I
eventually write around Mesh4x.

Cheers.

[Juan Marcelo Tondato]

The previous scenario should be possible to support in mesh4x. Currently j2me version of mesh4x supports sinchronize a record in the mobile phone with the cloud or with other mobile phone via sms.
The cloud supports add a new item via http too.

Related with multi table sync, check my last changes, I am working in "Issue 113:  Add to adapters the possibility to sync a mesh group (group of datasets)".
Currently you can sync a lot of tables (see ) in the same sync session.

[Juan Marcelo Tondato]

Sorry, see org.mesh4j.sync.adapters.composite.HibernateMultiTableTests.

In this test I am synchronizing a lot of tables froma  db with a MsExcel sheets.

There are other examples in the same package with MsAccess tables, MsExcel sheets and a lot of feed xml files.

 

Thanks in advance.
JMT

[amsmota]

I'm getting very frustrated wit this, certainly because my lack of
knowledge about feeds, atom, rss, rdf and the likes.

I put up a simple example using SyncEngine with a HibernateSyncAdapter
as source and a MQAdapter as target, that the only thing that does is
to send a item.getContent().getPayload().asXML() to the Queue.

What I got in the Queue is this:

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:expense_report="http://meridianp2p/Tables/expense_report#"
xmlns:owl="http://www.w3.org/2002/07/owl#" xmlns:xsd="http://
www.w3.org/2001/XMLSchema#"><expense_report:expense_report
rdf:about="uri:urn:3000130176"/></rdf:RDF>

I was expecting to see the actual data to be synchronized somewhere in
the payload, but it seems it's not the case.

What am I doing, or assuming, wrong here?

Cheers.

[amsmota]

Thankfully, it seems that I had tempered with the code too much, as
after I reverted all my changed Mesh4j code I can now see the correct
data. I have some more problems now, but let me play with this some
more (although I have to present my case tomorrow to make a go/no-go
decision and I'm yet very uncomfortable)...

[Juan Marcelo Tondato]

Please check my changes (synchronize with svn).

 

For create a db adapter use HibernateSyncAdapterFactory.

thanks in advance,

JMT

[jmt]

A simple example to sync a db table with a rss xml file:

// create a sync adapter based on hibernate
ISyncAdapter adapterSource =
HibernateSyncAdapterFactory.createHibernateAdapter(
"jdbc:mysql:///mesh4xdb", // url connection
"root", // user
"", // password
com.mysql.jdbc.Driver.class,
org.hibernate.dialect.MySQLDialect.class,
"mytable", // table name
"http://mesh4x/test", // base url for rdf, put null is
you decide dont use rdf and use plain xml
"c:\\test", // folder for mapping
files
NullIdentityProvider.INSTANCE);

// create a rss xml file adapter
String fileName = "c:\\test\\Myfeed.xml";
Feed feed = new Feed("test", "test", "");
ISyncAdapter adapterTarget = new FeedAdapter(fileName,
NullIdentityProvider.INSTANCE, IdGenerator.INSTANCE,
RssSyndicationFormat.INSTANCE, feed);


// sync
SyncEngine syncEngine = new SyncEngine(adapterSource, adapterTarget);
List<Item> conflicts = syncEngine.synchronize();


// asserts
Assert.assertNotNull(conflicts);
Assert.assertTrue(conflicts.isEmpty());

List<Item> sourceItems = syncEngine.getSource().getAll();
List<Item> targetItems = syncEngine.getTarget().getAll();
Assert.assertEquals(sourceItems.size(), targetItems.size());

ISyncAdapter targetAdapter = syncEngine.getTarget();
for (Item sourceItem : sourceItems) {
Item targetItem = targetAdapter.get(sourceItem.getSyncId());
Assert.assertTrue(sourceItem.equals(targetItem));
}

[jmt]

RDF is used to standardize the data to a common definition, you can
consume the same information from different application expressed in
different formats using the rdfs definition of the data set, for
example Xform, PlainXML, rdf.

If you dont need this transformations you do not need to use RDF, you
must use plain xml.

[jmt]

I hope that those examples can help you.

Example for sync a group of db tables:

ISyncAdapter adapterOpaque = new InMemorySyncAdapter("opaque",
NullIdentityProvider.INSTANCE); // can be any ISyncAdapter

String[] tables = new String[]{"myTableName1", "myTableName2"};

ISyncAdapter adapterSource =
HibernateSyncAdapterFactory.createSyncAdapterForMultiTables(
"jdbc:mysql:///mesh4xdb",
"root",
"",
com.mysql.jdbc.Driver.class,
org.hibernate.dialect.MySQLDialect.class,
tables,
"http://mesh4x/test", // base url for RDF, put
null if you decide dont use RDF
"c:\\test",
NullIdentityProvider.INSTANCE,
adapterOpaque); // all items that are not
match with tableName1 or tableName2 are saved on that adapter

// Sync example
ISyncAdapter adapterTarget = new other sync adapter .....

SyncEngine syncEngine = new SyncEngine(adapterSource, adapterTarget);

....

[jmt]

To understand kindness of RDF we can see the following sync server
(public mesh4x test server), not change the information (is of other
user tests).
Oswego is a table of a MsAccess file, and we use EpiinfoClient to
create the mesh group in the server and sync the data, EpiinfoClient
is using the core mesh4x lib (the same code of my previous example).

See following urls, and view page source:

http://sync.staging.instedd.org:8080/mesh4x/feeds/Epiinfo/Oswego/schema
http://sync.staging.instedd.org:8080/mesh4x/feeds/Epiinfo/Oswego/schema?content=xform

http://sync.staging.instedd.org:8080/mesh4x/feeds/Epiinfo/Oswego/mappings

http://sync.staging.instedd.org:8080/mesh4x/feeds/Epiinfo/Oswego?format=rss20

http://sync.staging.instedd.org:8080/mesh4x/feeds/Epiinfo/Oswego?format=atom10

http://sync.staging.instedd.org:8080/mesh4x/feeds/Epiinfo/Oswego?format=kml

http://sync.staging.instedd.org:8080/mesh4x/feeds/Epiinfo/Oswego?content=xform

http://sync.staging.instedd.org:8080/mesh4x/feeds/Epiinfo/Oswego?content=plain_xform
http://sync.staging.instedd.org:8080/mesh4x/feeds/Epiinfo/Oswego?content=plain_xml

[jmt]

Example of creation of a HttpSyncAdapter for sync data to the Sync
Server:

HttpSyncAdapterFactory.createSyncAdapter("http://
sync.staging.instedd.org:8080/mesh4x/feeds/Epiinfo/Oswego", new
LoggedInIdentityProvider());

[Juan Marcelo Tondato]

Some others ISyncAdapter creation examples :)

Example of creation of sync adapter for MSAccess table:

   MsAccessSyncAdapterFactory factory = new MsAccessSyncAdapterFactory(
               "c:\\test",                                                                      // directory for hibernate mapping files
               "http://sync.staging.instedd.org:8080/mesh4x/feeds");     // base url for RDF, put null if you decide dont use RDF
   String mdbFileName = "c:\\test\myMDB.mdb";
   String tableName= "myTableName";
   ISyncAdapter singleTable = factory.createSyncAdapterFromFile(tableName, mdbFileName, tableName, new LoggedInIdentityProvider());

 

Example of creation of sync adapter for MSAccess tables:

   MsAccessSyncAdapterFactory factory = new MsAccessSyncAdapterFactory(
               "c:\\test",                                                                      // directory for hibernate mapping files
               "http://sync.staging.instedd.org:8080/mesh4x/feeds");     // base url for RDF, put null if you decide dont use RDF

   String[] tables = new String[]{"tableName1", "tableName2"};
   ISyncAdapter adapterOpaque =  new any sync adapter...
   String mdbFileName = "c:\\test\myMDB.mdb";
   ISyncAdapter multiTable = factory.createSyncAdapterForMultiTables(mdbFileName, tables, new LoggedInIdentityProvider(), adapterOpaque);

 

Example of creation of sync adapter for MsExcel sheet:

   MsExcelRDFSyncAdapterFactory rdfExcelFactory = new MsExcelRDFSyncAdapterFactory("http:\\mesh4x\example") ;       // base url for RDF
   ISyncAdapter adapterMsExcelXLSrdf = rdfExcelFactory.createSyncAdapter("c:\\test\\myExcel.xls", "sheetName", "idColumnName", new LoggedInIdentityProvider());
   ISyncAdapter adapterMsExcelXLSXrdf = rdfExcelFactory.createSyncAdapter("c:\\test\\myExcel.xlsx", "sheetName", "idColumnName", new LoggedInIdentityProvider());

   MsExcelRDFSyncAdapterFactory plainXmlExcelFactory = new MsExcelSyncAdapterFactory();
   ISyncAdapter adapterMsExcelXLS = plainXmlExcelFactory .createSyncAdapter("c:\\test\\myExcel.xls", "sheetName", "idColumnName", new LoggedInIdentityProvider());
   ISyncAdapter adapterMsExcelXLSX = plainXmlExcelFactory .createSyncAdapter("c:\\test\\myExcel.xlsx", "sheetName", "idColumnName", new LoggedInIdentityProvider());

   ISyncAdapter adapterOpaque =  new any sync adapter...
   Map<String, String> sheets = new HashMap<String, String>();
   sheets.put("sheet1", "Code");
   sheets.put("sheet2", "Code");
   sheets.put("sheet3", "Code");
   ISyncAdapter adapterMsExcelMultiSheets = createSyncAdapterForMultiSheets("c:\\test\\myExcel.xlsx", new LoggedInIdentityProvider(), sheets, opaqueAdapter);

Example of creation of Folder adapter:

    ISyncAdapter adapterFolder = FolderSyncAdapterFactory.createFolderAdapter("c:\\test\myFolderToSync", new LoggedInIdentityProvider()), IdGenerator.INSTANCE);

 

Example of creation of KMLAdapter:

    ISyncAdapter adapterKML = KMLDOMLoaderFactory.createKMLAdapter("c:\\test\myKml.kml", new LoggedInIdentityProvider());
    ISyncAdapter adapterKMZ = KMLDOMLoaderFactory.createKMLAdapter("c:\\test\myKml.kmz", new LoggedInIdentityProvider());

 

Example of dummy adapter:
   ISyncAdapter adapterDUMMY = new InMemorySyncAdapter("an adapter name", new LoggedInIdentityProvider());

 

Example of GoogleSpreadsheets adapter:
   GoogleSpreadSheetSyncAdapterFactory gssPlainXMLFactory= new GoogleSpreadSheetSyncAdapterFactory();
    ISyncAdapter adapterGss = gssPlainXMLFactory.createSyncAdapter("userName", "passWord", "spreadsheetName", "sheetName", "idColumName", null, new LoggedInIdentityProvider(), "sheetName");
  

  GoogleSpreadSheetRDFSyncAdapterFactory  gssRDFFactory = new GoogleSpreadSheetRDFSyncAdapterFactory(("http:\\mesh4x\example") ;       // base url for RDF
  ISyncAdapter  adapterGssRDF = gssRDFFactory.createSyncAdapter("username", "password", "spreadsheetName", "sheetName", "idColumnName", "lastUpdateColumnName", new LoggedInIdentityProvider, "spreadsheetName");

Any feedback is welcomed.

Thanks in advance,

JMT

[Juan Marcelo Tondato]

Currently, we can classify adapters in 3 groups:

1. Adapters based on tables with structured data:
      DB adapters (mysql, msaccess), Excel adapter, GoogleSS adapter.
      RDF is required here, adapters need to interpret the information (xml content).
      take values from xml elements and put each value in typed fields (column in a db table, cell in excel).

2. Adapters with a own xml definition:
     Folder adapter (define the mechanism to represent a file and a file content).
     KML/KMZ adapter (each element of the kml file is management in a special way, placemarks, Styles, etc)

3. Opaque Adapters:
    Adapters that store items and not interpret the xml content.
    Cloud Adapter (Sync server), S3Adapter (amazon s3 adapter), Feed adapter (rss, atom)

Compatibility between adapter types:

    Adapters type 1 can be synchronized with Adapters type 1 or type 3.
    Adapters type 2 can be synchronized with Adapters type 2 or type 3.
    Adapters type 3 can be synchronized with all types.
 
But, Adapters type 1 can not be synchronized with adapter type 2 (for the moment ;))

 

We can define another adapter type, subclass of type 1 (structured data table), when you define this kind of adapters using plain xml (no rdf), in this case the xml content is interpreted as literal strings and it is impossible to run conversion of types. In this case you can synchronize this adapter with the same adapter in the other side, with opaque adapters, but if you synchronize with other adapter type (for example excel vs mysql), the synchronization can be fail for incompatibility of data types.