Offline support

[Sergio AFANOU]

Hi all,

I'm trying to add more fonctionality to lipsync. For example, I'm trying to handle offline modifications.

Here is my idea : 

Sometimes, you modify a file but you don't have an internet connexion available to instantly sync that modification. I know that lsyncd tries several times to sync the modification if the first try fails. But what if you shutdown the computer or just kill the process ? Your modification is lost.

So I'm digging into lsyncd, and playing with the configuration file to be able to store any modification (and it's details) into a database to be able to resume from where the process has stopped.

Do you guy's know a way to read/write from/into a sql lite database with bash ?

Thanks.

---

Sergio

[mart]

I don't know for bash with sqlite but I've two idea (for what it's
worth) :
- if you start adding several "advance" functionalities, shouldn't we
switch to a more evolved programming language (C, python,...) but I'm
not an expert in bash so maybe we can achieve that without switching
of language.
- what about storing something (info file, the diff in the file,...)
in the ~/.lipsyncd folder ?

But I agree ofline sync is really useful.

[Anthony Goddard]

Both ideas make sense to me, perhaps the offline solution could use a
local cache directory initially and then a more advanced db solution
later as the needs evolve.

[Sergio AFANOU]

Hi,

Storing informations in ~/.lipsync directory is a good idea. The info I need to store is about the event (create, delete, modify, attribute), the source, the target, the remote server, and some other info like that. I need to be able to spawn the uncompleted events on lipsync startup/restart.

@Anthony : I don't know about local cache. Can you tell me more about that ?

If we choose to switch language, we should choose Python or Perl as they are closer to bash. Bash scripters can tell us more about that.

---

Sergio

[Anthony Goddard]

Hi, 

I didn't mean any specific cache, just meant caching it locally, hidden from the user, so ~/.lipsync/cache would be the example in this case. I'm just referring to the temporary store location when the machine is offline as cache.

Cheers,

Anthony

[Sergio AFANOU]

Thanks Anthony.

I found a package sqlite which is able to write and read info from a lightweight database.

The user needs to install it (in addition to the other packages).

Does it sound reasonable ?

---

Sergio

[martin trigaux]

We should always try to reduce the number of dependencies but sqlite
is a pretty basic package imho. So what should be stored in the
database ?

If we want to just remember the files to update, we don't need much.
Maybe just a file like :
# list of files to update
+ mynewfile1.txt
- myuselessfile1
+ mynewfile2.jpg
- folder/uselessfile2
* updatededfile.odt
...

We can have a cache folder but why store the files as we don't do
incremental backup like dropbox (or maybe we will ?) and store
timestamps in the sqlite database.

About the programming language, maybe think also if this software has
to be ported to other platforms than linux (I don't know if it's
planned), bash definitely won't work on windows (I don't know about
mac).

[tdrusk]

I feel like you are making this too complicated.
What is wrong with doing an if statement
if ping to the server works then sync the files.
if not, save a list of files and their modification date+time to a
text file. When the connection comes back up check if the files on the
server have been modified after the files you have previously
modified. If not, sync it and delete the text file.

[Conor Schaefer]

I'm very much with tdrusk on this. Part of the appeal of Lipsync, at least at this stage, is that it is very lightweight. Given its heavy reliance on bash, working with text files should be sufficiently reliable, and favorably efficient. The offline syncing feature is a notable addition that should definitely be implemented, once a reasonable method is devised.

[Sergio AFANOU]

Hi Guys !

Well, I thought about a database (just a file where I can write some structured info) for many reasons.

First, I thought about writing infos into a simple text file, then I realized that there may be some issues retrieving and parsing correctly the stored data. The info I need to store is about the event (create, delete, modify, attribute), the source, the target, the remote server, and some other info like that. I'm affraid it will be more complex to correctly parse the data in a plaintext file than to retrieve them from a sqllite.db file.

Second, It is more simple to modify parameters with a GUI if you just read them from a database and store the new value. I don't know how applications in Linux read an write configuration data. We can dig that way (but lipsync can become less portable on other platforms like Mac Os).

Third, I choosed sqllite because It is portable and lightweight. The file inside can be read with any sqlite client. I'm currently looking for a linux sqlite client.

I also thought about doing a simple if statement as tdrusk suggested. I couldn't develop further because when lsyncd issues a file synchronization (spawn the event), we don't have access to the process (or the code) to make our connexion test. So I have been obliged to develop level 3 scripts to implement the offline stuff. At that level, I can control what to do if a file needs to be synced.

Feel free to comment :)

Cheers !

---

Sergio

[Balazs Pocze]

2011/8/18 Sergio AFANOU <afanou...@gmail.com>:

> Hi Guys !
> Well, I thought about a database (just a file where I can write some
> structured info) for many reasons.
> First, I thought about writing infos into a simple text file, then I
> realized that there may be some issues retrieving and parsing correctly the
> stored data. The info I need to store is about the event (create, delete,
> modify, attribute), the source, the target, the remote server, and some
> other info like that. I'm affraid it will be more complex to correctly parse
> the data in a plaintext file than to retrieve them from a sqllite.db file.
> Second, It is more simple to modify parameters with a GUI if you just read
> them from a database and store the new value. I don't know how applications
> in Linux read an write configuration data. We can dig that way (but lipsync
> can become less portable on other platforms like Mac Os).
>
> Third, I choosed sqllite because It is portable and lightweight. The file
> inside can be read with any sqlite client. I'm currently looking for a linux
> sqlite client.
> I also thought about doing a simple if statement as tdrusk suggested. I
> couldn't develop further because when lsyncd issues a file synchronization
> (spawn the event), we don't have access to the process (or the code) to make
> our connexion test. So I have been obliged to develop level 3 scripts to
> implement the offline stuff. At that level, I can control what to do if a
> file needs to be synced.
> Feel free to comment :)
> Cheers !

The SQLite client is simple, just use sqlite package, and then you can
run the db via command line (or from a shell script).
But in fact I didn't know well the lipsync, but when i read the source
it seems to using rsync, not? The rsync can reach itself up from any
time.

--
"Overhead, without any fuss, the stars were going out."