[Hype] Multiple wiki sync thanks to couchdb

[Danielo Rodríguez]

Hello,

This post is just to gather some attention to the development of the couchadaptor for TW.

Some months ago, I spend some time trying to understand couchdb and pouchdb, which was a frustrating task (FAIL!). Some days ago I suddenly check it again and now I understand it perfectly (I can't understand why).

With this suddenly appeared knowledge I checked the couchadaptor for TW that wshallum wrote. I set it up, and I saw its great potential. I made a couple of changes to fit my particular needs (which have been merged now into the main repo) and now I have two sets wikis synced. I have to say that this is amazing, and the developer is very open to add new functionality. In fact, in just two days he has merged all my pull requests and have implemented himself some of my suggestions, so expect cool things from this!!

We have plans to implement an easier way to set this up, but if anyone wants to test it right now I can assist him in the process. In fact, when all this is mature I want to write a small manual, or let the original author do himself if he is interested.

This has many possible implementations, not all of them are perfect right now, and some scenarios like offline working would need a different approach, but:

•   Sync multiple files to a centralized store of tiddlers. The way I'm using this right now is keeping an standalone version (html file) on my phone, manually saving it through tiddlyfox to have a local copy, and allow the syncadaptor synchronize the tiddlers. I have another copy of the file (same html file, but with less tiddlers) that gets synced with the online database. 
  
•   Sync different databases to make different wikis share tiddlers. Currently I don't know how to determine which sub-set of tiddlers you want to sync, but cool enough.
  

This is EXTREMELY BETA. The reason I'm talking about it here is because I want the talented people that visit this group to raise their point of view, ideas, questions and maybe collaborating to the development.

I found CLOUDANT to be the best choice to test this out. They have a very good free plan and accept CORS request by default (needed for working with a standalone version of TW)

Regards.

[Danielo Rodríguez]

Another posible implementation I thought about:
Blog usage. I'm not sure how it works, but couchdb allows you to set a database for public read only access. If that is possible and thanks to the synchronization you can have a TW wit full editing capabilities where you can add content and save it to the dB. Then the read only wiki just access to those information and displays it, without needing to save the file each time or edit it before publishing.

[Jean-Philippe Rutault]

Hi,

Just testing, some sync problem with tw5 open on two computers, but it seems to work well.

Another great usage I see of your implementation is for managing attachment inside tw5 (image, pdf, ...). For now, i just put an Iframe of futon inside a tiddler which allow me to upload attachment to /tw/images for example. Still investigating.

But TW5 couple with couchdb seems to be a great idea.

Thanks

[Danielo Rodríguez]

Hello Jean

El miércoles, 20 de mayo de 2015, 11:39:19 (UTC+2), Jean-Philippe Rutault escribió:

Hi,

Just testing, some sync problem with tw5 open on two computers, but it seems to work well.

Could you describe the problems? How heavy was the usage?

 

Another great usage I see of your implementation is for managing attachment inside tw5 (image, pdf, ...).

Just to be clear, I'm not the author of the plugin, I'm just collaborating on its development.

 

For now, i just put an Iframe of futon inside a tiddler which allow me to upload attachment to /tw/images for example. Still investigating.

I really like your idea. Please keep us aware of your findings. 

[Jean-Philippe Rutault]

Very light usage, still testing and investigating this architecture. I have TW5 opens on two computers. I just do basic editing on one, then refresh the wiki on the other. I ended up with Draft tiddler with sync problem and was unable to remove them. I add to go to the couchdb interface to remove the corresponding document (Draft of New Tiddler).

Indeed, I can't install any plugins, neither via drag'n'drop of the plugins tiddlers, nor via the new control panel menu. It ends up asking indefinitely to reload the wiki to have the new plugin to take effect. Add to remove manually the corresponding tiddler.

Despite all this problems, i find this approach very interesting since it kept all revisions of tiddlers and allow the management of attachment directly inside couchdb in the same document as the document used to store the tiddler. Very promising for beta.

[Danielo Rodríguez]

Hello Jean.Philippe,

El miércoles, 20 de mayo de 2015, 15:02:59 (UTC+2), Jean-Philippe Rutault escribió:

Very light usage, still testing and investigating this architecture. I have TW5 opens on two computers. I just do basic editing on one, then refresh the wiki on the other. I ended up with Draft tiddler with sync problem and was unable to remove them. I add to go to the couchdb interface to remove the corresponding document (Draft of New Tiddler).

Do you let the Draft of New Tiddler be saved on both wikis? Probably the problem is that the Draft Exist already on the database, and then you tried to create a new Draft on another wiki. Normally when you finish editing a draft it is removed from the DB. As you leaved the draft in draft mode you have ended with it permanently stored on the DB. Because it is a generic Draft, TW does not tries to load it from the database, so the revision is never updated. It's a small problem, but fixable. Thanks for reporting.

 

Indeed, I can't install any plugins, neither via drag'n'drop of the plugins tiddlers, nor via the new control panel menu. It ends up asking indefinitely to reload the wiki to have the new plugin to take effect. Add to remove manually the corresponding tiddler.

This is a known issue. It's better if you do it the other way round: 

1. pick an empty tw, 
   
2. install all the plugins you want/need
   
3. Reload TW
   
4. Add the couchdb plugin, 
   
5. use that TW (upload it to your database if necessary)
   

Maybe we can provide a mechanism in the plugin to upload the modified version of the wiki (without any tiddler, just plugins) to the DB.

 

[Jean-Philippe Rutault]

Still playing and have a new little problem. Perhaps you can explain to me if it's a normal or if it's a sync bug.

1. First I create a tiddler called TestJpeg for example with the type image/jpeg and the empty field _canonical_uri
2. I save the tiddler and upload an attachment to the corresponding document in couchdb.
3. I edit my TestJpeg tiddler and add the full url of the uploaded attachment in _canonical_uri field then save and all works well. I can now transcluide my tiddler image as a normal tiddler image, but...

If I edit my Tiddler, even if I make no modifications, save or cancel the editing, the document in couchdb looses its attachment. It now appears as _attachments [object object] and anew field _attachments with [object object] appears in my tiddler. No more transclusion, no more attachment in my tiddler.

Thanks in advance

[Danielo Rodríguez]

Hello Jean,

I don't think this is a problem with the adapter itself but the way you are using couch-db. Could you clarify me some points?

2. I save the tiddler and upload an attachment to the corresponding document in couchdb

What is the corresponding document? The tiddler you just have created? If so, that may be the problem. I don't know how editing a document will affect to its attachments, I have to investigate it.

Anyway, why don't you just create a image tiddler? Like normal drag and drop? I'm not forcing you to do it that way, I want to know which advantages you see with your method.

[Jean-Philippe Rutault]

The corresponding document is the tiddler I just created in couchdb.

No immediate advantages just playing to see what I can and can't do. I think it could be interesting to store lots of attachments in couchdb linked to a specified tiddler, without having those attachments directly in tiddlywiki.

Anyway thanks for your quick answer.

--
You received this message because you are subscribed to a topic in the Google Groups "TiddlyWiki" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tiddlywiki/pZzttVrh5Kg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/db7fa5bf-af3b-4367-a92b-4a7d9a5322c7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jean-Philippe Rutault]

In fact one big advantage, I can embed huge pdf file without problem with this technique (tested with two PDF of 50Mo and tiddlywiki continue to work fluently, with the drag'n'drop technique, the first one crashes tiddlywiki). To sumarize

1. I create a tiddler named My Huge Pdf for example, with the type application/pdf and a _canonical_uri field
2. In Futon, i upload the huge PDF as an attachment to the document My Huge Pdf wich represent my newly created tiddler
3. Copy the link location of the uploaded PDF and add it to the _canonical_uri field
4. Save my tiddler -> The attachment disappear and a new_field is created in my tiddler
5. Re_edit my tiddler to get rid of the new auto-generated field _attachment. Save
6. Re-upload the huge pdf and now all works well as long as I didn't re-edit my tiddler ;)

Hope that was clear. So it would be great if the attachment doesn't disappear in step 4.

To unsubscribe from this group and all its topics, send an email to tiddlywiki+unsubscribe@googlegroups.com.

To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.

[Danielo Rodríguez]

Hello Jean,

El jueves, 21 de mayo de 2015, 20:10:55 (UTC+2), Jean-Philippe Rutault escribió:

In fact one big advantage, I can embed huge pdf file without problem with this technique (tested with two PDF of 50Mo and tiddlywiki continue to work fluently, with the drag'n'drop technique, the first one crashes tiddlywiki).

With your previous message I understood your intentions. But it was not clear for me if the browser manages differently an image within a tiddler (blob) and a link to an image. With your pdf example seems pretty clear that is better to just use the URI. I think this is because we avoid TW core processing the blob body. I'm curious, what couchdb provider are you using? Or is it a local couchdb? 

 

To sumarize

1. I create a tiddler named My Huge Pdf for example, with the type application/pdf and a _canonical_uri field
2. In Futon, i upload the huge PDF as an attachment to the document My Huge Pdf wich represent my newly created tiddler
3. Copy the link location of the uploaded PDF and add it to the _canonical_uri field
4. Save my tiddler -> The attachment disappear and a new_field is created in my tiddler
5. Re_edit my tiddler to get rid of the new auto-generated field _attachment. Save
6. Re-upload the huge pdf and now all works well as long as I didn't re-edit my tiddler ;)

Hope that was clear. So it would be great if the attachment doesn't disappear in step 4.

You have been clear. I'll investigate following your steps. I have a couple of theories... 

[Jean-Philippe Rutault]

I use a local couchdb install (v1.6.1) on archlinux.

[Danielo Rodríguez]

Hello Jean,

I am not able to reproduce what you are saying. Here are my steps:

• I open the wiki and create a new tiddler of type image/png
• I add the field _canonical_uri
• I save the tiddler, and let it be synced to the couchdb
• Then, I add an attachment and I add the canonical_uri value directly on the db interface
• On the next sync cycle the image is displayed correctly on TW
• No matter what I do to the tiddler everything works perfectly.

Just some seconds later....

Oook, now I get what have happened. I have read your instructions again and I tried again. I just removed the _attachment field and the image has gone. That's because I have removed the attachment field from the tiddler, and it is synced back to the server. Then couchdb receives a new version of the doc without any attachment, so it removes the attachment. As long as you keep the _attachment field everything should went fine.

I think the attachment field should not be included in the tiddler, because it has nothing to do with the tiddler itself. I'll check if its fine to not sync that field to TW. Meanwhile, just keep that field ;)

[Danielo Rodríguez]

Ooops,

Nothing happens until I reload the wiki. I'll check to fix this issue. 

I don't know why that field is being included in the tiddler, since only the fields under the fields field (sorry for this ) should be included, any other field should be ignored.

[Danielo Rodríguez]

On my way back to home I realized what is the real issue: All the tiddler upload to the server are PUT requests. This creates a new document on couchdb instead of updating it with new values. Since the uploaded document does not have the attachment it is lost.

The solution of course is to make POST request for tiddlers that are just updated. I will open a ticket for that.

[Danielo Rodríguez]

According to couchdb documentation:

Note that the "stub":true attribute denotes that this is not the complete attachment. Also, note the length attribute added automatically. When you update the document you must include the attachment stubs or CouchDB will delete the attachment.

So we have to do is to properly manage the attachment field 

[Danielo Rodríguez]

Here is the github ticket for fixing the attachment problem:

https://github.com/wshallum/couchadaptor/issues/11#issuecomment-104886226

[Jean-Philippe Rutault]

Thanks a lot. That's exactly the sort thing I was guessing. I'll continue playing and only add to say that this is a great implementation of tiddywiki sync since it allow now revision and attachment management.

[Danielo Rodríguez]

El sábado, 23 de mayo de 2015, 16:20:01 (UTC+2), Jean-Philippe Rutault escribió:

Thanks a lot. That's exactly the sort thing I was guessing. 

Hello Jean,

The attachments problem have been fixed on this branch of the project:

https://github.com/wshallum/couchadaptor/tree/attachments-issue-fix

If you are able to test the plugin using that branch before I merge it to the master one I will be very grateful.  

I'll continue playing and only add to say that this is a great implementation of tiddywiki sync since it allow now revision and attachment management. 

Please don't hesitate to continue commenting.  Your feedback is very valuable and thanks to you we have fixed a bug. So if you find more problems or think about new features, please let us know.

[Danielo Rodríguez]

Hello Jean,

I have opened this ticket on GitHub to discuss the problem with several instances running at the same time. This particular tickets is about deletions sync.

https://github.com/wshallum/couchadaptor/issues/15

If anyone has an idea about how to handle this, I'm very pleased to hear about it. Here is the ticket text:

When you have two wikis that syncs to the same couchdb newly created tiddlers are correctly synced between two, but deletions does not.

I don't know how to understand this as a feature or as a problem. There is only one scenario where I'm totally sure this is a problem: in Draft management. Drafts are temporal by definition and once you have finished edditing you don't want that draft anymore, and of course you don't want a draft to be kept on several instances of TW.

To reproduce the Draft problem:

• Open a tiddlywiki that syncs to a couchdb in two different computers or two different tabs.
• Edit a tiddler or create a new tiddler and leave it in draft mode long enough to be synced
• Wait in the other instance of the wiki to the next "skinny tiddlers" fetch
• Draft appears on recent tab, absolutely empty unless you open it.
• Finish editing on the first wiki. Draft is deleted from DB
• Open the draft on the second instance. You have an empty draft, this can cause confusion because you also have the normal tiddler.
• Try to finish editing on the second instance with the empty draft and you will get an error because that document does not exist on the database anymore

I'm not sure how to address this problem. Avoid draft synchronization should not be an option, because I strongly thing that drafts should be synced.

[Jean-Philippe Rutault]

Hello Danielo,

all seems to work correctly. My attachments no longer disappeared upon tiddlers edition. Thanks.

So, I came upon a new bug. Once a tiddler has an attachment in couchdb, it is impossible to rename it since you went into sync problem, no matter the attachment was used or not.

1. Create a tiddler with some text.
2. In futon upload an attachment to the newly created tiddler.
3. Come back to tiddlywiki, edit the tiddler and modify the title to rename it. Save.
4. You get a  sync error and the tiddler and its attachment disappeared from couchdb, but you still see it in Tiddlywiki.
5. Reload Tiddlywiki, the tiddler with sync problem no longer exist.

Indeed, thanks very well.

I also manage to create a custom template embedding all the plugins I need in couchdb and all seems to work very well.

[Danielo Rodríguez]

Hello Jean,

Thank you very much for your valuable feedback. I'm not sure how to face this new problem, but I'll get fun trying. I was afraid this could happen anyway.

What do you mean with "template with all the plugins". I suppose you mean an html file containing all the plugins already installed.

[Jean-Philippe Rutault]

If you find a solution to this new problem this would be great. Anyway, renaming a tiddler with attachment didn't occur so often I guess, so it's not an annoying bug. But surely a personal challenge to resolve.

Yes template with all the plugins means html file containing all the plugins.