Re: hi, I want to join the conduit development

[fpemud]

On 03/01/2013 01:53 AM, Alexandre Rosenfeld wrote:

Hi there,

There hasnt been any more activity in Conduit for a while. My project to make it work as a daemon was never integrated in Conduit since the mantainers stopped it's development soon after my summer of code finished. The code dump for my project can be found here (http://code.google.com/p/google-summer-of-code-2009-gnome/downloads/detail?name=Alexandre_Rosenfeld.tar.gz). I have no idea if it still fits the master branch, but you can try.

I always belived the main interface for Conduit was terrible and it was best if it was a system service instead. My project was somehow successfull, you can run Conduit without the GUI (you always could in a sense, but I fixed some problems with that).

But I also believe that the code base has rotten a lot, since many of it's dependencies have released new versions which might be incompatible now (PyGObject comes to mind).

I'm still interested in the idea of what Conduit does, but I'm not sure the code in it's current state is up to it. If you really plan to move it forward I can help you, either with the current code or the concepts behind it.

Regards,
Alexandre

Alexandre Rosenfeld

On Sun, Dec 30, 2012 at 10:59 AM, fpemud <fpe...@gmail.com> wrote:

hi, I'm a software engineer. I need a sync software on linux.
After some comparison, I think conduit suits me best. But I'd like
conduit be a daemon, and i have some other requirements.

In this page: https://live.gnome.org/Conduit/SummerOfCode/Projects
I found there was already a "conduit daemon project" in 2009. Is this
project finished or dead?
I want to continue this project. Any people here can give me some guide?

Really happy to meet you! Your help is critical to me!
Is it easier to get you by this new email: alexandre...@gmail.com ?

I have read some code of conduit, but I'm sorry the real work will not start recently.

Would you please help me make clear some concept of conduit project?
I think it's important to go in the right direction.

Below is my understanding of conduit project. Please correct me if I'm wrong.

I think the core philosophy of conduit is "Sync anything between any device through any media".
All of the other sync program (unison, dropbox, etc) can only synchronize files.
Although in UNIX everything is a file, we can't take this view in the area of synchronizing.
It will be great to build an open ecosystem around conduit.
The current code has already implemented the "sync anything" part by a plug-in system.

Conduit should play a role of "sync back-end", the current mainline code mixes back-end and front-end together, so "conduit daemon project" is a big step by splitting them up.
There should be many front-end, the current configuration ui (conduit-cfg-ui) should be one of them.
Some front-end may be embedded in application, like an Evolution plugin.

Administrator could create sync-group to sync system data.
Normal user could also create sync-group to sync his/her data. (how to deal priviledge? Require the user has same uid on both machines?)

Conduit should be decentralized and it already is.
(Can you tell me how does conduit implements it?)

Sync-group could be persistent or one-time.
one-time sync-group could be the back-end of "NFC sync" application.

Conduit could be used to do live-migration.
If we develop a virt-machine-live-migration plugin in which utilizes like libvirt, it can sync VM between desktop PCs.
We could also develop a plugin to do process-live-migration by Checkpoint/Restart.

Some type of sync could need application's co-operation.
Like If I live-migrate a game to another machine while the game is playing, the game should save it's state to a buffer, and conduit should start the game on the other end, sync the buffer, and restore game state by this buffer.


That all in my head now.
These are mainly about what conduit should / should not do. Conduit is a framework, so I think these are important.
Please correct me if I'm wrong, thanks!

[Alexandre Rosenfeld]

Really happy to meet you! Your help is critical to me!

Is it easier to get you by this new email: alexandre...@gmail.com ?

Any email is fine by me.

 

I have read some code of conduit, but I'm sorry the real work will not start recently.

Would you please help me make clear some concept of conduit project?
I think it's important to go in the right direction.

Below is my understanding of conduit project. Please correct me if I'm wrong.

I think the core philosophy of conduit is "Sync anything between any device through any media".
All of the other sync program (unison, dropbox, etc) can only synchronize files.
Although in UNIX everything is a file, we can't take this view in the area of synchronizing.
It will be great to build an open ecosystem around conduit.
The current code has already implemented the "sync anything" part by a plug-in system.

Take a look at https://live.gnome.org/Conduit/Development, it has a lot of documentation about how the Conduit source code is architectured. Take a look at those links as well, the DataSource, etc, they will help understand the core concepts behind Conduit.

 

Conduit should play a role of "sync back-end", the current mainline code mixes back-end and front-end together, so "conduit daemon project" is a big step by splitting them up.
There should be many front-end, the current configuration ui (conduit-cfg-ui) should be one of them.
Some front-end may be embedded in application, like an Evolution plugin.

That was my project's final objective. I spent a lot of time making all of the dataproviders independent of the UI by making a abstraction of the configuration dialog. This means there is no longer any Gtk code in the dataproviders and mostly of the code that starts them also have no dependency on Gtk.

Conduit is mostly ready to run as a daemon, but one main issue is that it's written in Python and inherits it's threading issues.

For instance, Conduit and it's data providers all run in the same process, which means a bug in one dataprovider can affect any other synchronization that is being done by Conduit. 

And Python's threading model is very difficult to handle well, especially when you start working with bindings (something to do with locks when Python code calls C code, as far as I remember).

 

Administrator could create sync-group to sync system data.
Normal user could also create sync-group to sync his/her data. (how to deal priviledge? Require the user has same uid on both machines?)

Not sure if I understood, but please never tie a user data to his uid, it will create a bunch of other problems.

 

Conduit should be decentralized and it already is.
(Can you tell me how does conduit implements it?)

As I said, the dataproviders all run in the same process, so it's not quite as decentralized as it should.

 

Sync-group could be persistent or one-time.
one-time sync-group could be the back-end of "NFC sync" application.

Conduit could be used to do live-migration.
If we develop a virt-machine-live-migration plugin in which utilizes like libvirt, it can sync VM between desktop PCs.
We could also develop a plugin to do process-live-migration by Checkpoint/Restart.

Some type of sync could need application's co-operation.
Like If I live-migrate a game to another machine while the game is playing, the game should save it's state to a buffer, and conduit should start the game on the other end, sync the buffer, and restore game state by this buffer.


That all in my head now.
These are mainly about what conduit should / should not do. Conduit is a framework, so I think these are important.
Please correct me if I'm wrong, thanks!

It's mostly correct what you said, I'll just try to explain what I learned from Conduit.

First, it's more then just Dropbox because it understands what kind of data each party can handle. This is really awesome, since you can synchronize all kinds of metadata, such as ratings, play count, date taken, etc, for each kind of data. And it can also convert from one format to another, such converting videos to the right size or converting from wma to mp3.

Second, it connects applications and services without them ever knowing about each other. This is amazing, for instance you can hook your Shotwell library to any kind of online publishing, without having to wait for the Shotwell team to implement it. I began using Conduit to sync my iPod with my Rhythmbox library.

So here comes the difficult part, since it can sync virtually anything, it just depends on what dataproviders you write for it, it becomes quite hard to test it and especially to create a good user interface to it.

I believe Conduit should focus on a few use cases and do it right, even if it means not doing everything possible. 

For instance, each application already has the best interface for each data type, we shouldn't have to reimplement it in Conduit. What we need is to put Conduit inside each application.

One case which Conduit would be amazingly good at is to implement a system similar to the Android's share button. Click Share in an application and Conduit would present to you a list of dataproviders which would accept it.

But technically, Conduit is not up to the task. My vision for it would be to have a lightweight DBus daemon written in Vala that would spawn a process for each dataprovider, which would also communicate via DBus and could be implemented anyway it wishes (preferably inside the application that owns that dataprovider).

Anyway, I have nothing concrete as well, just a few ideas.