RaspberryPi and TiddlyWeb

[PMario]

@FND @chris,
and everyone else, who is interested :)

I did start a project because of this post: "Alternative for saving
TW"
[1] http://groups.google.com/group/tiddlywiki/browse_thread/thread/d875a3282c807d8a

Just to see if it would be simple enough to throw some (cheap)
hardware at the TW save problem. ... and get some sort of scalability,
without using the cloud.

Reading the TiddlyWiki group, people want to have files. Being it
TiddlyWiki.html or one file per tiddler. They don't trust databases,
__if they have to manage them on there own.__

* As chris pointed out in [1], TiddlyWebWiki would be the easiest
option.
But It creates a single TW per instance. Having a look at the RasPi
CPU and memory usage to get things done, imo 2 or 3 instances at the
same time would be possible.

* IMO Hoster is the next option.
Installing it is allmost the same, but possibilities are much more
powerful. The UI doesn't do well enough (for me) but that can be
fixed.

So I choose hoster as the basis installation.
The store is "textstore", because people want files, and the RasPi
can't lift MySQL anyway.
Hoster gets a much simpler UI (to be done)

Step 1:
======
The goal is a single user multi TiddlyWiki system connected to a home
network.
Revisions are done by tiddlyWeb
tiddlers are stored at the RasPi's SD card.
The UI can create new TiddlyWiki's
The UI can share bags between TWs
TiddlyWiki knows, how to transfer tiddlers from one bag to an other.

Step 2:
======
same as Step 1
Multi user Multi TW system
Revisions are done by tiddlyWeb
A new UI for hoster that can handle several users.
Storing the tiddlers on a harddisk should be an option.
Public and private tiddlers are possible
still connected to a home network only

No inclusion mechanism like in TiddlySpace. If you want to have this
function just use TiddlySapce :)

Step 3:
======
same as Step2
Revisions are done by git.
git may be used to push stuff to the web
git may be used to pull stuff from the web
git can be used to share stuff

Step 4
=====
A lot if ideas here ....


what do others think?

have fun!
mario

[chris...@gmail.com]

On Sun, 13 Jan 2013, PMario wrote:

> Just to see if it would be simple enough to throw some (cheap)
> hardware at the TW save problem. ... and get some sort of scalability,
> without using the cloud.

Awesome that you are pursuing this.

> * As chris pointed out in [1], TiddlyWebWiki would be the easiest
> option.
> But It creates a single TW per instance. Having a look at the RasPi
> CPU and memory usage to get things done, imo 2 or 3 instances at the
> same time would be possible.

As I've said several times a tiddlywebwiki installation can have as
many tiddlywiki's as desired, just by creating recipes.

So instead of using hoster you could have pursued the option of making
it easy to create recipes. The advantage there would be that
tiddlywebwiki + <a recipe tool> would have a smaller footprint than
hoster. You would, though, lose quite a few features that hoster
provides.

> * IMO Hoster is the next option.
> Installing it is allmost the same, but possibilities are much more
> powerful. The UI doesn't do well enough (for me) but that can be
> fixed.

As with pretty much everything I've created that has a frontend on it,
the frontend is entirely there to provide a place for someone else to
improve. So my hope, all along, with hoster, has been that people
would make better looking and better functioning UI.

> Step 2:

I can guess some of the steps you have planned here:

> Multi user Multi TW system

change registration system in hoster

> Revisions are done by tiddlyWeb
> A new UI for hoster that can handle several users.

I'm not sure what you mean here, as the UI is already multi-user?

> Storing the tiddlers on a harddisk should be an option.

using diststore so some of the bags are on the hard drive

> Public and private tiddlers are possible

this is already possible in hoster. hoster has the concepts of public,
private and proctected for both recipes and bags, as well as editable
policies.

> No inclusion mechanism like in TiddlySpace. If you want to have this
> function just use TiddlySapce :)

hoster has a form of inclusion that is _far_ more powerful than what
is used in tiddlyspace: You can favorite bags and then include them in
recipes. This means you can choose functionality from across the
system with greater granularity than tiddlyspace provides.

So I would say if you're going to pursue using hoster, make sure you
really dig around in the system to see all that it can do. Good luck
and please keep posting about your progress.
--
Chris Dent http://burningchrome.com/
[...]

[FND]

> Awesome that you are pursuing this.

Indeed - I'm watching this with great interest.

> Step 3: [...] Revisions are done by git.

I've made some progress on the Git-based store[1], but it's a bit more
work than I'd anticipated - might take a few more days, perhaps even
weeks.


-- F.


[1] https://github.com/FND/tiddlywebplugins.gitstore

[PMario]

On Jan 13, 9:54 pm, chris.d...@gmail.com wrote:
> As I've said several times a tiddlywebwiki installation can have as
> many tiddlywiki's as desired, just by creating recipes.

And how can you make them all look different, without a web-management
layer? Imo command line and text editors are no option.

> So instead of using hoster you could have pursued the option of making
> it easy to create recipes. The advantage there would be that
> tiddlywebwiki + <a recipe tool> would have a smaller footprint than
> hoster. You would, though, lose quite a few features that hoster
> provides.

right. I think it is easier to go top down, than bottom up here. Et
least for me :)

> As with pretty much everything I've created that has a frontend on it,
> the frontend is entirely there to provide a place for someone else to
> improve. So my hope, all along, with hoster, has been that people
> would make better looking and better functioning UI.

I think it works well, but it is too complicated for newbies. Imo
that's why TiddlySpace was made.

> > Multi user Multi TW system
>
> change registration system in hoster

Just a button

> > Public and private tiddlers are possible
>
> this is already possible in hoster. hoster has the concepts of public,
> private and proctected for both recipes and bags, as well as editable
> policies.

I know, but configuration is quite complicated and not visible
enough.

> hoster has a form of inclusion that is _far_ more powerful than what
> is used in tiddlyspace:

I know and I'm probably the only user, who uses it with the teamwork
place [1].

[1] http://hoster.peermore.com/recipes/TeamWork/tiddlers.wiki#HowToJoinTeamWork
This video was the easiest way to explain it. (and I struggled
somewhere near the end :)

> So I would say if you're going to pursue using hoster, make sure you
> really dig around in the system to see all that it can do. Good luck
> and please keep posting about your progress.

yup

-m

[Andrew Back]

On 13 January 2013 21:14, FND <FN...@gmx.net> wrote:
>> Awesome that you are pursuing this.
>
> Indeed - I'm watching this with great interest.

+1; I love the idea of a TiddlyWeb appliance, and a RasPi-based
solution is much more pragmatic than the one we took with TiddlyCubed
(which, shamefully, didn't get very far).

>> Step 3: [...] Revisions are done by git.
>
> I've made some progress on the Git-based store[1], but it's a bit more
> work than I'd anticipated - might take a few more days, perhaps even
> weeks.

A Git store would be very cool, and is pretty much the reason why I
used ikiwiki for a few sites. I like being able to edit files locally
and then push to update a remote wiki, and all the other benefits of
working with your stuff in a DVCS.

Cheers,

Andrew

--
Andrew Back
http://carrierdetect.com

[FND]

/me waves at Andrew

I assume you came here via some sort of Google Alerts notification?

> A Git store would be very cool, and is pretty much the reason why I
> used ikiwiki for a few sites. I like being able to edit files locally
> and then push to update a remote wiki, and all the other benefits of
> working with your stuff in a DVCS.

FWIW, I've just written up my rationale/vision here:
http://fnd.tiddlyspace.com/gitstore


-- F.

[Andrew Back]

On 14 January 2013 07:34, FND <FN...@gmx.net> wrote:
> /me waves at Andrew

Hey Fred!

> I assume you came here via some sort of Google Alerts notification?

No, I'm subscribed. I must admit I don't read messages that often but
this thread piqued my interest.

>> A Git store would be very cool, and is pretty much the reason why I
>> used ikiwiki for a few sites. I like being able to edit files locally
>> and then push to update a remote wiki, and all the other benefits of
>> working with your stuff in a DVCS.
>
> FWIW, I've just written up my rationale/vision here:
> http://fnd.tiddlyspace.com/gitstore

Ah, great stuff. I love the idea of being able to use lots of little
RasPi or similar hosts and federating across them. As you know, this
was part of the plan with TiddlyCubed, which I still think has a lot
of mileage and I wish I had the time and necessary skills to move it
forward.

Best,

[PMario]

Any ALPHA testers ?
-m

[Måns]

Hi Mario

> Any ALPHA testers ?

Yes please. Let's try it out.
Have you created an IMG for easy/quick installation on a sd card?
- Or a guide for a hoster installation on raspian?

Great initiative :-)

Cheers Måns Mårtensson

[PMario]

https://github.com/pmario/tweb-config

have fun!
PS: It works best with a "vanilla raspbian" image and a class 10 SD
card

[PMario]

On Jan 24, 12:32 am, Måns <humam...@gmail.com> wrote:
> Yes please. Let's try it out.
> Have you created an IMG for easy/quick installation on a sd card?
> - Or a guide for a hoster installation on raspian?

Image is cracy. Several hundred MByte. My upload rate is meeeh .. :)

I did create a shell script, that should make steps simpler.
See the Readme at https://github.com/pmario/tweb-config I did those
steps with a vanilla image, as I wrote the howto. So it should work.

To create the script I created a lot of snippets. Step by step. Which
I'll put up on tiddlyspace. But this needs some refactoring too.

I hope you like the UI :)

have fun!
mario

[PMario]

Some general info.

The UI automatically creates 2 types of recipes, when you create a
TiddlyWiki
* public
* private

This system is similar to TiddlySpace. See similar. I don't want to
run TS on the RasPi. My focus / intention is different.

There are 3 types of bags
* private .. owner can see and edit
* protected .. everyone can see, owner can edit
* public .. everyone can see and edit

==========

The public TW contains
* bag protected
* bag public

The private TW contains
* bag protected
* bag public
* bag private

in this order ... which defines the tiddlers visibility.

===========

This is the exact same concept, that hoster uses. So the configuration
is very flexible. With the web interface, there will be the
possibility to create bags with more granular adjustments.

==========

Next TODOs to finish Step I:
* The UI can share bags between TWs
* TiddlyWiki knows, how to transfer tiddlers from one bag to an other.
* Make the snippest public

Step II
* IMO is 20% way done.

Step III
* Test FND's gitstore. ...

============

let me know, what you think.

have fun!
-mario pietsch

[Måns]

PS: It works best with a "vanilla raspbian" image and a class 10 SD
card

I use the the SD card for bootup only.
My SD card is a 16 Gb miniSD (from an old smartphone) which I have partitioned into 4 partitions, one of them is used for booting different OS'es from usb.
The Berry Boot Installer is very handy for doing netinstall of several systems (multiboot) onto usb-disk etc.
It also seems to increase system speed very much.

Cheers Måns Mårtensson

[Måns]

Hi Mario

Great job!!!
Thanks for explaining things :-)

I will need a little time to be able to test it. (The weekend).

I will return later (with a lot of questions - I guess ..)

Cheers Måns Mårtensson

[Lauren Horn]

I have a small stable of 'Pis, and would love to kick the tires.

On Wed, Jan 23, 2013 at 12:50 PM, PMario <pmar...@gmail.com> wrote:

Any ALPHA testers ?

-m

--
You received this message because you are subscribed to the Google Groups "TiddlyWeb" group.
To post to this group, send email to tidd...@googlegroups.com.
To unsubscribe from this group, send email to tiddlyweb+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/tiddlyweb?hl=en.

[PMario]

On 24 Jan., 02:03, Måns <humam...@gmail.com> wrote:
> The Berry Boot Installer <http://www.berryterminal.com/doku.php/berryboot>is

> very handy for doing netinstall of several systems (multiboot) onto
> usb-disk etc.

Looks interesting.

I use 3SD cards for OS switching atm.

1 x 8GB(class10), 2 x 4GB(class4) where I did the time measurement.
The class 10 performance will be with the snippets :)

-m

[James Savage]

Mario,
I already had raspbian installed using the berry installer so I thought I would give this a go. I started by cloning the raspbian instal as a starting point for tweb.

I like the config utility approach - in fact I think we need something similar for tiddlyspace on Ubuntu to reduce the initial learning needed for self hosting tiddlyspace.

Your Readme instructions have cd without a target - this confused me but I thought it was worth continuing...
You will recognise the following steps.  I have commented to show what I found.
 

On your Pi

• cd        Note.  I was only working on the pi so I was already in /home/pi.
  
• sudo tweb-config OK

• follow the steps 1-6
• 2 .. OS upgrade .. 20 min (class 4 SD card) OK
• 3 .. Install TWeb dependencies .. 7 min OK
• 4 .. New system user www-tweb .. 1 min OK
• 5 .. Install TiddlyWebWiki .. 6 min OK
• 6 .. Login as www-tweb .. 20 seconds Here I had trouble

1. The instructions displayed on screen after step 6 are:

-- su www-tweb OK  (asks for password as expected)and then returns to command line silently


--cd  Ah what do I do here?  I am still in /home/pi

--twinstance tweb && cd tweb Here I get an error
Traceback File "/usr.local/bin/twinstance", line 28, in <module>
status = not main(sys.argv)
.....further traceback omitted
Then I get a
OSError Permission denied: '/home/pi/tweb'     It would appear that tweb has not been created....

Are you able to see where I hay have gone wrong or misunderstood the instructions?

Kind regards,

James

[PMario]

Thanks for the feedback!

---

The cd <enter> command is there for a reason
It changes to the home directory of the active user :)

cd ~
would do the same

twinstance tweb && cd tweb It needs to be typed exactly as I wrote
it.

is the same as

twinstance tweb <enter>
cd tweb <enter>

[PMario]

On Jan 28, 1:20 pm, James Savage <consult...@gmail.com> wrote:
> Then I get a
> OSError Permission denied: '/home/pi/tweb'     It would appear that tweb
> has not been created....
>
> Are you able to see where I hay have gone wrong or misunderstood the
> instructions?

Yes.

I did push a slightly modified (hopefully improved) version to github.
https://github.com/pmario/tweb-config

as user pi@raspberrypi in the home directory
wget https://raw.github.com/pmario/tweb-config/master/tweb-config

will update it

have fun!
mario

[James Savage]

Mario,
thanks for that.  That cd second was critical :-)  You learn something every day... The remaining steps were straightforward and the server came up OK and renders a templated empty tw.
I expected the Site Title of the empty tw to be set to the name that I had given the tw but its easy to change and maybe better not to force the names to be identical.

One suggestion to make the result more friendly would be to allow a shorter URL if recipe and bag is not specified 
eg http://192.168.1.96:8080/wiki_name

Now all I need to do is find a use for the pi with this capability:-)

Well done.

Kind regards,
James

[chris...@gmail.com]

On Mon, 28 Jan 2013, James Savage wrote:

> One suggestion to make the result more friendly would be to allow a shorter
> URL if recipe and bag is not specified
> eg http://192.168.1.96:8080/wiki_name

Here's the code for a quick plugin that does that:

https://gist.github.com/4664082

[PMario]

On Jan 28, 11:12 pm, James Savage <consult...@gmail.com> wrote:
> One suggestion to make the result more friendly would be to allow a shorter
> URL if recipe and bag is not specified
> eghttp://192.168.1.96:8080/wiki_name

nope.

Not in the first run. There is some content, that is not part of the
alpha atm.

The idea for "Step I" is, to add more information to the different
URLs

eg:
http://myIP:8080/
.. will contain a welcome page that lists your existing "recipes /
TiddlyWikis" + links to open them. With a long URL for the start.
AND .. it will tell you a little bit about TiddlyWeb structure.
Shorting the URL at this early stage, the new user will miss a lot of
important information. You will miss a lot in the future, and limit
your possibilities.

http://myIP:8080/recipes/
.. will tell the user some info about recipes + a shortcut to the wiki
page

http://myIP:8080/recipes/default/tiddlers
.. a bit more info about the different tiddler serialisations + some
info for possible developers, that may want to use the JSON
representation

http://myIP:8080/recipes/default/tiddlers.wikis
.. the TiddlyWiki representation

http://myIP:8080/
.. may get a link to the "manager" wiki, that tells you how to do URL
shortening, in a way that fits to "Step II - Multi User: Multi TW". It
will create users, new TiddlyWikis and probably more ....

===

> Now all I need to do is find a use for the pi with this capability:-)

Mine will get a harddisk and will serve TiddlyWiki pages

Pages, that are TWs
Pages, that are plain HTML displaying markdown tiddlers ....
Pages, that show pictures

=====

but first some more feedback would be needed, to make it more user
friendly.

have fun!
mario

[Andrew Back]

Hi Mario,

On 24 January 2013 00:27, PMario <pmar...@gmail.com> wrote:
> https://github.com/pmario/tweb-config

Handy script!

As requested off-list in response to a blog post on
TiddlyWeb/TiddlWiki on RasPi [1], here is some feedback:

With the direction the script is taking at present I think it would be
great to have a pre-prepared Raspbian image with this loaded and that
has unnecessary packages removed, useful things like Avahi
(Zeroconf/mdns) installed, TiddlyWeb pre-installed and configured to
start at boot, and perhaps a selection of useful plug-ins
pre-configured or available to install from the menu. In short: a
TiddlyWeb/TiddlyWiki Raspberry Pi "appliance" image.

The tweb-config script should really be in /usr/local. Only binaries
that are installed/upgraded/removed by a package manager should be in
/usr/bin. Sym linking to files in /home is even worse as this can be a
filesystem that is mounted upon login and thus not available system
wide.

There are quite a lot of steps and I'm not convinced of the merit of
listing TiddlyWeb users before creating our own (we know there will
only be one) and then checking again.

If the script were to be used as the configuration menu for a
"TiddlyAppliance" it would be useful to have a system update option.
However, if it is to be used with a Raspbian install where there may
be other apps installed I would question the logic of adding Linux
system management commands to an application configuration menu. If
you're at the command line and using sudo you should know how to
"apt-get upgrade" etc.

Cheers,

Andrew

[1] http://designspark.com/eng/blog/building-a-personal-microcontent-server-with-raspberry-pi

[PMario]

Hi Andrew,
Thanks for the feedback.

On Feb 5, 9:58 am, Andrew Back <and...@carrierdetect.com> wrote:
> With the direction the script is taking at present I think it would be
> great to have a pre-prepared Raspbian image with this loaded and that
> has unnecessary packages removed, useful things like Avahi
> (Zeroconf/mdns) installed, TiddlyWeb pre-installed and configured to
> start at boot, and perhaps a selection of useful plug-ins
> pre-configured or available to install from the menu. In short: a
> TiddlyWeb/TiddlyWiki Raspberry Pi "appliance" image.

At the very beginning, I thought about this possibility. The latest
Raspian image ist about 483MB. To create an image, that would work out
of the box, I'd need to create the whole image on my own. eg: the
raspi-config script only pops up the first time you start a fresh
installation and it modifies /etc/ system files. This initial boot
behaviour is needed, because users need to setup at least there
keyboard layout. At the second boot you need to call raspi-config on
your own.

I'd need to reset the initial boot behaviour, after creating a TWeb
setup.

> The tweb-config script should really be in /usr/local. Only binaries
> that are installed/upgraded/removed by a package manager should be in
> /usr/bin. Sym linking to files in /home is even worse as this can be a
> filesystem that is mounted upon login and thus not available system
> wide.

You are right I'll move the script to /usr/local.

At the beginning (alpha), I'll stick with the /home symlink. I mounted
the "rpi home" directory with sshfs to my laptop. This makes it
possible, to edit with a propper editor and push it to home/pi@rpi
very fast.

> There are quite a lot of steps and I'm not convinced of the merit of
> listing TiddlyWeb users before creating our own (we know there will
> only be one) and then checking again.

It's a simple check, if someone, started the twanager server outside
the TW instance directory (eg: www-tweb/tweb). In this case there will
be no user at all. The order is probably a relict prior to the "user
is inside instance directory" test. I'll need to check this.

> If the script were to be used as the configuration menu for a
> "TiddlyAppliance" it would be useful to have a system update option.

At the moment this is part of the "sudo tweb-config" menu. "5 Install
the TiddlyWebWiki system". User needs to be root.

> However, if it is to be used with a Raspbian install where there may
> be other apps installed I would question the logic of adding Linux
> system management commands to an application configuration menu. If
> you're at the command line and using sudo you should know how to
> "apt-get upgrade" etc.

Right.
If I personally type "sudo apt-get update && sudo apt-get upgrade" I
allways forget the -y. I'll probably move the low level system
commands into a submenu.

have fun!
mario

[PMario]

The beta INTRO at TiddlyWiki group:
https://groups.google.com/group/tiddlywiki/browse_thread/thread/2a52728e9b64a75c

have fun!
mario