Adding tiddlers from the command line with curl

[Dirk-Jan Swagerman]

Dear all,

I seem to struggle a bit on how to use curl to add content to my
tiddlyweb over http.

a) Adding a binary tiddler with
curl -X PUT -H 'Content-Type: image/png' --data-binary
@<pathtojpg>.jpg <mywebsite.com>/bags/common/tiddlers/test

works nicely without any errors

b) Adding a binary tiddler with
curl -X PUT -H 'Content-Type: application/json' -d t":"hello"}'
<mywebsite.com>/bags/common/tiddlers/monkey

gives the error:
403 Forbidden: (UserRequiredError(u'for bag default: real user
required to create',),)

I have several questions
1) I have the following policy:
{"read": [], "create": ["R:ADMIN"], "manage": ["R:ADMIN"], "accept":
["R:ADMIN"], "write": ["R:ADMIN"], "owner": "administrator", "delete":
["R:ADMIN"]}

So i would expect that not everyone is allowed to write a tiddler to
my website. So why is a) allowed and b) not? This seems an error in
the authentication system?

2) Can anyone provide me with an example/pointer to documentation on
how to use curl to pass on username & passwork (i use the cookie_form
authentication). I tried -u<username:password> and combinations of -b
and -c but no luck so far.

3) An example of how to use curl to upload a simple text file
including the method for user authentication would also be
appreciated. I tried
curl -X PUT -H 'Content-Type: text/plain' -data @<filename>
<mywebsite.com>/bags/default/tiddlers/monkey

which resulted in the error message:

TiddlerFormatError: bad headers in tiddler: monkey, need more than 1
value to unpack.

Sorry if this clearly documented somewhere but i was unable to find it
so far.

Best regards,
Dirk-Jan

[FND]

> 1) I have the following policy:

> [...]

> So i would expect that not everyone is allowed to write a tiddler to
> my website. So why is a) allowed and b) not? This seems an error in
> the authentication system?

I cannot seem to reproduce this; spun up a fresh TiddlyWebWiki instance,
modified the "common" bag's policy as described and curl'd:
$ curl -v -X PUT -H 'Content-Type: image/jpg' --data-binary \
@bunny.jpg http://localhost:8080/bags/common/tiddlers/image
Results in a 403, as expected.

> 2) Can anyone provide me with an example/pointer to documentation on
> how to use curl to pass on username & passwork (i use the cookie_form
> authentication).

> [...]

> 3) An example of how to use curl to upload a simple text file
> including the method for user authentication would also be
> appreciated.

It's been a while since I've done this myself, so I hope others will
jump in to help out.

> I tried curl -X PUT -H 'Content-Type: text/plain' -data @<filename>
> <mywebsite.com>/bags/default/tiddlers/monkey
> which resulted in the error message:
> TiddlerFormatError: bad headers in tiddler: monkey, need more than 1
> value to unpack.

Do you want to PUT that file as a binary or as a regular tiddler?
With text/plain, TiddlyWeb will try to deserialize the text format
(headers, blank line, body):
modifier: FND
tags: foo bar baz

lorem ipsum
dolor sit amet
http://tiddlyweb.peermore.com/wiki/recipes/docs/tiddlers/text%20tiddler

(aside: Chris isn't around this week, so it might take a couple of days
for him to respond.)

-- F.

[Dirk-Jan Swagerman]

Hi,

Thanks for your reply. My intention is to find a way to import
tiddlers (both binary images and text based tiddlers) using the
command line (i try to set up something where i can email an image of
a poster or text, run some ocr and add both the image and the
recognized text as a tiddler in my tiddlywiki). Actually twanager
would be an option for me too and your brief example seems to work for
me with twanager.

When i use any of the following commands (where tt is a text file
containing your example). The tiddler is not added and i basically get
the HTML source of tiddlywiki back in the command line.

curl -X PUT -H 'Content-Type: text/plain' -data-binary @tt

<mywebsite.com>/bags/default/tiddlers/monkey
curl -X PUT -H 'Content-Type: text/plain' -data @tt <mywebsite.com>/
bags/default/tiddlers/monkey

Thanks for your help and suggestions. Still would like to get some
clarity on the curl examples even though i know now how to use
twanager for a text tiddler. May need to wait for Chris then?

Best regards,
Dirk-Jan

On Feb 17, 10:28 am, FND <F...@gmx.net> wrote:
> > 1) I have the following policy:
> > [...]
> > So i would expect that not everyone is allowed to write a tiddler to
> > my website. So why is a) allowed and b) not? This seems an error in
> > the authentication system?
>
> I cannot seem to reproduce this; spun up a fresh TiddlyWebWiki instance,
> modified the "common" bag's policy as described and curl'd:
>      $ curl -v -X PUT -H 'Content-Type: image/jpg' --data-binary \

>          @bunny.jpghttp://localhost:8080/bags/common/tiddlers/image

> Results in a 403, as expected.
>
> > 2) Can anyone provide me with an example/pointer to documentation on
> > how to use curl to pass on username & passwork (i use the cookie_form
> > authentication).
> > [...]
> > 3) An example of how to use curl to upload a simple text file
> > including the method for user authentication would also be
> > appreciated.
>
> It's been a while since I've done this myself, so I hope others will
> jump in to help out.
>
> > I tried curl -X PUT -H 'Content-Type: text/plain' -data @<filename>
> > <mywebsite.com>/bags/default/tiddlers/monkey
> > which resulted in the error message:
> > TiddlerFormatError: bad headers in tiddler: monkey, need more than 1
> > value to unpack.
>
> Do you want to PUT that file as a binary or as a regular tiddler?
> With text/plain, TiddlyWeb will try to deserialize the text format
> (headers, blank line, body):
>      modifier: FND
>      tags: foo bar baz
>
>      lorem ipsum

>      dolor sit amethttp://tiddlyweb.peermore.com/wiki/recipes/docs/tiddlers/text%20tiddler

[FND]

> i try to set up something where i can email an image of a poster or
> text, run some ocr and add both the image and the recognized text as
> a tiddler in my tiddlywiki

Sounds exciting.

> your brief example seems to work for me with twanager

That's good to know (so it's not a typo in the bag name, for example).

> The tiddler is not added and i basically get the HTML source of
> tiddlywiki back in the command line.

Hm, I have no idea what might cause the response text to be a TiddlyWiki
under these circumstances.
Are you using any plugins other than tiddlywebwiki?

Could you try the following reduced test case, just so we can rule out
any external factors:
$ mkdir /tmp/tweb && cd /tmp/tweb # just being extra paranoid here
$ twinstance foo
$ cd foo
$ twanager server localhost 8080 & # terminate later with kill %1
$ echo -e "foo: bar\n\nlorem ipsum `date`" > myTiddler
$ curl -v -X PUT -H "Content-Type: text/plain" \
--data-binary @myTiddler \
http://localhost:8080/bags/common/tiddlers/myTiddler

> curl -X PUT -H 'Content-Type: text/plain' -data-binary @tt
> <mywebsite.com>/bags/default/tiddlers/monkey
> curl -X PUT -H 'Content-Type: text/plain' -data @tt <mywebsite.com>/
> bags/default/tiddlers/monkey

It's safer to always use --data-binary, as curl does some weird magic
with the --data option.
(Also note the double dash, the lack of which I now think might explain
your issues.)

> Still would like to get some clarity on the curl examples

Assuming the above works, the outstanding issue is authentication with
curl? I haven't had time to look into that myself yet, but I'll get
around to it eventually...
I think it involves a POST username & password to /challenge/cookie_form
and then using curl's cookie jar on subsequent requests.

-- F.

[Dirk-Jan Swagerman]

Thanks! Your suggestion to look at a post of user & password to curl
worked:

1) First use curl to post username and password to the cookie_form and
store a cookie for it locally:
curl ---cookie-jar cookies.txt -d
"user=<username>&password=<password>&submit=submit" <yourwebsite.com>/
challenge/cookie_form

2) Pass the created cookie to subsequent post to upload tiddlers
curl --cookie cookies.txt -X PUT -H 'Content-Type: text/plain' --data-
binary @<filename> <yourwebsite.com>/bags/default/tiddlers/monkey

So, problem was indeed the single dash. I cannot reproduce my original
issue where i could upload a jpg without being properly authenticated
any more. So all issues closed! Again thanks for your help!

Best regards,
Dirk-Jan

[chris...@gmail.com]

On Sun, 21 Feb 2010, Dirk-Jan Swagerman wrote:

> Thanks! Your suggestion to look at a post of user & password to curl
> worked:

For sake of completeness, what I tend to do when I want to populate a
remote TiddlyWeb is use TiddlyWebWeb, which provides a local tiddlyweb
against which you can run twanager commands but for which the store is
a remote tiddlyweb. This is especially useful with TiddlyWeb running
on google app engine.

There's more info on both of those things in these blog posts:

http://cdent.tumblr.com/post/278948050/smooth-tiddlyweb-on-app-engine
http://cdent.tumblr.com/post/283065885/tiddlywebweb-to-app-engine

If you are using TiddlyWeb based authenticate with users stored in the
tiddlyweb store, then you can put authentication credentials in the
config of the tiddlywebweb.

If you've got something that's working for you, I recommend you stick
with it, but if you are curious to play around in the guts of
TiddlyWeb stuff, tiddlywebweb can shine some light.

--
Chris Dent http://burningchrome.com/~cdent/
[...]

[FND]

> Your suggestion to look at a post of user & password to curl worked

Great - I've added it to the docs:
http://tiddlyweb.peermore.com/wiki/bags/docs/tiddlers/How%20can%20I%20authenticate%20using%20curl%3F

-- F.

[FND]

> I tried -u<username:password> [...] but no luck so far.

I've just tried this basic auth approach, and it seems to work fine:
curl -X PUT -H 'Content-Type: application/json' \
-d '{ "recipe": [["alpha", ""]] }' \
-u <username>:<password> \
http://localhost:8080/recipes/omega

That gets you around the issue of having to grab a cookie first.

-- F.