Re: cammount issue, android app and general dev-* cleanup questions

[Mathieu Lonjaret]

Hi,

Please take my answers with a grain of salt.
Brad will correct later when I'm wrong.

On Tue, Oct 16, 2012 at 6:40 AM, Bill Thiede <couch...@gmail.com> wrote:
> Pardon me for putting this all in one email, I can fork off the topics if
> one gets noisy.

it's fine like that, we'll open separate issues in the tracker if need be.

> So I'm experimenting with camlistore trying to understand how it works, and
> apparently I have some magic power for breaking things. First thing I
> noticed is that dev-cammount is not setting some required environment
> variables. I added the following to make it work for me:
>
> +$ENV{"CAMLI_DEV_KEYBLOBS"} = "$Bin/config/dev-client-dir/keyblobs";
> +$ENV{"CAMLI_AUTH"} = "userpass:camlistore:pass3179";
> +$ENV{"CAMLI_KEYID"} = "26F5ABDA";
> +$ENV{"CAMLI_SECRET_RING"} = "$Bin/pkg/jsonsign/testdata/test-secring.gpg";
>
> I have no doubt this is simple bitrot, and wouldn't be surprised if the
> other dev-* helper scripts had similar omissions. Would a CL putting all the
> "$ENV{...} ="s into a common module and having all the dev-* scripts import
> it be welcome?

it works for me out of the box but it's probably because I have a
working client config file, so I don't need all the env vars. I had
stopped using the dev-* scripts since then.
What about dev-camput, is it working properly for you without any change?

> Secondly, I got cammount mounting, and for some reason the first thing I
> tried caused a panic. All I did was run dev-server in one terminal,
> dev-cammount in another and then ran the following:
>
> $ cat /tmp/cammount-dir/sha1-280c3ea4d9f9b31975211ae669630aa1b3ea30f0
>
> Which happens to be the permanode from "Publish root node for dev-pics-root"
> in the Web UI. I made this small change:
>
> --- a/pkg/schema/filereader.go
> +++ b/pkg/schema/filereader.go
> @@ -185,7 +185,7 @@ func NewFileReader(fetcher blobref.SeekFetcher,
> fileBlobRef *blobref.BlobRef) (*
>
> func (ss *Superset) NewFileReader(fetcher blobref.SeekFetcher)
> (*FileReader, error) {
> if ss.Type != "file" && ss.Type != "bytes" {
> - return nil, fmt.Errorf("schema/filereader: Superset not of
> type \"file\" or \"bytes\"")
> + return nil, fmt.Errorf("schema/filereader: Superset not of
> type \"file\" or \"bytes\": %q", ss.Type)
> }
> size := int64(ss.SumPartsSize())
> return &FileReader{
>
> And can see that ss.Type == "permanode", which explains the panic.
> nodeReader looks like this:
>
> type nodeReader struct {
> n *node
> ss *schema.Superset // not nil, always of type "file" or "bytes"
> }
>
> And I'm not sure if that comment is saying other code should be enforcing
> that guarantee, or if the original author expected it would never be set to
> anything else in this code?

Ok, I can reproduce that. I'll have a closer look, thanks.

> Third and lastly, the android app doesn't seem to work, it is looking for
> URLs that are not served by the dev-server configuration. Is there an
> example of a configuration that works with the android app? If only the
> URLs have been renamed, and the functionality still exists to support the
> android client, someone can point me to the CL for the new layout?

No idea about that, sorry.

> Bill
>
> PS If it's worth creating a HOWTO for getting all the various dev-* parts
> working together, to tour through the various functionalities that
> camlistore provides, I wouldn't mind taking a stab at it. But I don't want
> to put in too much effort if people think it should be self-evident, or it
> is assumed that people capable of contributing to camlistore at this stage
> would be able to infer 'how' from looking at the source.

I believe the names are explicit and simple enough that what the
commands generally do should be self-evident. Then running them (e.g
dev-camput) without argument should print the help for the command
behind the scenes (resp. camput) and that should be enough for the
user/dev to go on from there.
That said, it's all a work in progress, so some of those helps are
probably incomplete.

[Mathieu Lonjaret]

Btw, here's an example on how to give cammount a try (assuming you
have an alright config setup):

mkdir foo
cp somefiles foo
camput file -filenodes foo

this should end with an output like this:
2012/10/17 18:05:50 Waited on root node "foo/":
&{sha1-44d8c046d9f8d8878215f2a5360b961b398589f2 300 false}
sha1-44d8c046d9f8d8878215f2a5360b961b398589f2

then you can use cammount on the hash of the foo dir:
mkdir mnt
cammount mnt sha1-44d8c046d9f8d8878215f2a5360b961b398589f2

On Tue, Oct 16, 2012 at 6:40 AM, Bill Thiede <couch...@gmail.com> wrote:
> Pardon me for putting this all in one email, I can fork off the topics if
> one gets noisy.
>

> So I'm experimenting with camlistore trying to understand how it works, and
> apparently I have some magic power for breaking things. First thing I
> noticed is that dev-cammount is not setting some required environment
> variables. I added the following to make it work for me:
>
> +$ENV{"CAMLI_DEV_KEYBLOBS"} = "$Bin/config/dev-client-dir/keyblobs";
> +$ENV{"CAMLI_AUTH"} = "userpass:camlistore:pass3179";
> +$ENV{"CAMLI_KEYID"} = "26F5ABDA";
> +$ENV{"CAMLI_SECRET_RING"} = "$Bin/pkg/jsonsign/testdata/test-secring.gpg";
>
> I have no doubt this is simple bitrot, and wouldn't be surprised if the
> other dev-* helper scripts had similar omissions. Would a CL putting all the
> "$ENV{...} ="s into a common module and having all the dev-* scripts import
> it be welcome?
>

> Third and lastly, the android app doesn't seem to work, it is looking for
> URLs that are not served by the dev-server configuration. Is there an
> example of a configuration that works with the android app? If only the
> URLs have been renamed, and the functionality still exists to support the
> android client, someone can point me to the CL for the new layout?
>

[Mathieu Lonjaret]

> Secondly, I got cammount mounting, and for some reason the first thing I
> tried caused a panic.

That should be fixed now if you git pull.

[couch...@gmail.com]

On Wed, Oct 17, 2012 at 11:42:52AM +0200, Mathieu Lonjaret wrote:
> Hi,
>
> Please take my answers with a grain of salt.
> Brad will correct later when I'm wrong.
>
> On Tue, Oct 16, 2012 at 6:40 AM, Bill Thiede <couch...@gmail.com> wrote:
> > Pardon me for putting this all in one email, I can fork off the topics if
> > one gets noisy.
>
> it's fine like that, we'll open separate issues in the tracker if need be.
>
> > So I'm experimenting with camlistore trying to understand how it works, and
> > apparently I have some magic power for breaking things. First thing I
> > noticed is that dev-cammount is not setting some required environment
> > variables. I added the following to make it work for me:
> >
> > +$ENV{"CAMLI_DEV_KEYBLOBS"} = "$Bin/config/dev-client-dir/keyblobs";
> > +$ENV{"CAMLI_AUTH"} = "userpass:camlistore:pass3179";
> > +$ENV{"CAMLI_KEYID"} = "26F5ABDA";
> > +$ENV{"CAMLI_SECRET_RING"} = "$Bin/pkg/jsonsign/testdata/test-secring.gpg";
> >
> > I have no doubt this is simple bitrot, and wouldn't be surprised if the
> > other dev-* helper scripts had similar omissions. Would a CL putting all the
> > "$ENV{...} ="s into a common module and having all the dev-* scripts import
> > it be welcome?
>
> it works for me out of the box but it's probably because I have a
> working client config file, so I don't need all the env vars. I had
> stopped using the dev-* scripts since then.
> What about dev-camput, is it working properly for you without any change?

dev-camput does, as does dev-camget, dev-camsync, dev-pass-camput and
dev-camserver, or at least the will show the help. I guess those are the
important ones.

Problem dev wrappers:

$ ./dev-blobserver
Starting blobserver on http://localhost:3179/ in /tmp/camliroot-me/port3179/
2012/10/18 20:48:52 Using config file
/Users/me/Go/src/camlistore.org/config/dev-blobserver-config.json
2012/10/18 20:48:52 written config/selfgen_cert.pem
2012/10/18 20:48:52 written config/selfgen_key.pem
Error parsing config: error while configuring auth: Unknown auth type: ""

$ ./dev-camwebdav
./dev-camwebdav: line 3: ./build.pl: No such file or directory

$ ./dev-indexer
Failed to build camlistored at ./dev-indexer line 18.

$ ./dev-synctoindexer
./dev-synctoindexer: line 3: ./build.pl: No such file or directory

And googling for 'site:camlistore.org build.pl' doesn't show anything, so it
doesn't appear I missed any docs on creating my own build helper. It looks
like maybe the $ENV{} settings problem is localized to the dev-cammount, and
it wouldn't be that advantageous to move the env settings to a common file.
My initial concern was that at some point dev-camput would have similar env
bitrot as the build.pl in the other wrappers, but that might be moot, as
camput probably gets a lot of attention.

I will look into creating my own client config and not worrying about the
wrappers. Do you do just set the env vars in your .whateverrc or is there a
camlistore way of doing it? I see camput has an init verb, but the website is
pretty bare on it's documentation, in fact the code link @
http://camlistore.org/cmd/camput seems to be dangling.

Thanks for submitting a fix, and for giving me a pointer on mounting; I wasn't
using the -filenodes option before, and camput -h doesn't mention it. I'll
start reading the source to get a better understanding of what's possible.

Bill

[Brad Fitzpatrick]

Bill, welcome!

As you've noted, things are under construction.

Replies below....

Indeed.  Sorry, I guess I never got around to porting the old (unused) ones.  I'll clean those up tomorrow, or just delete/move them.

And googling for 'site:camlistore.org build.pl' doesn't show anything, so it
doesn't appear I missed any docs on creating my own build helper.

build.pl is no more.  It was a temporary Go build system before Go 1 was released with its own.  Camlistore went through a painful transition where our whole source tree and build system were turned all around.

I guess we didn't finish the transition in all places.

I will look into creating my own client config and not worrying about the
wrappers.  Do you do just set the env vars in your .whateverrc or is there a
camlistore way of doing it?  I see camput has an init verb, but the website is
pretty bare on it's documentation, in fact the code link @
http://camlistore.org/cmd/camput seems to be dangling.

I do all my development with the (working) ./dev-* scripts, using all their various flags (--wipe, etc), and I generally only develop in in-memory mode, not using MySQL or Mongo or anything.

My personal instance, though, runs under runsit (https://github.com/bradfitz/runsit --- undocumented, but those using it like it.  You could just as well use upstart or systemd or daemon-tools or whatever) and it has a runsit config file of:

$ cat /etc/runsit/camlihouse.json

{

 "user": "bradfitz",

 "cwd": "/tmp",

 "binary": "/home/bradfitz/bin/camlistored",

 "args": [

   "--configfile=/home/bradfitz/.camlistore/server-config.json"

 ],

 "ports": {

    "173.nn.nn.nn:443": "173.nn.nn.nn:443"

 }

}

(notable, no environment variables)

Where my server-config.json is:

{

        "baseURL": "https://xxxxxxxx.bradfitz.com",

"listen": "173.nn.nn.nn:443",

"TLS": true,

        "TLSCertFile": "/home/bradfitz/keys/camli-personal/ssl.crt",

        "TLSKeyFile": "/home/bradfitz/keys/camli-personal/ssl.key",

"auth": "userpass:bradfitz:xxxxxxx:+localhost",

        "identity": "6401800C",

"identitySecretRing": "/home/bradfitz/.camlistore/identity-secring.gpg",

"blobPath": "/mnt/backups/brad-camli-personal",

"mysql": "bradfitz@localhost:xxxxxxxxxxxx",

        "dbname": "yyyyyyyyyyyyy",

"replicateTo": []

}

But note that we're changing that config file slightly.  Mathieu renamed some fields just the other day... hopefully he'll go update all the docs too.  (hint hint ;))

That simple config file (which we call the "high-level config") should be enough for most people.  It gets converted into the raw low-level config file which has tons of knobs and is very verbose.  I think you can even mix the two styles in the config file and it'll partially merge them.  Or that was the plan.

Anyway, that's my dev & personal setup.

Feel free to pester us other places where the docs suck.  Getting started with a project is always the most painful part.

[Mathieu Lonjaret]

> I will look into creating my own client config and not worrying about the
> wrappers. Do you do just set the env vars in your .whateverrc or is there a
> camlistore way of doing it? I see camput has an init verb, but the website is
> pretty bare on it's documentation, in fact the code link @
> http://camlistore.org/cmd/camput seems to be dangling.

that's how I did it, but it's a bit awkward because I generated a gpg
keyring specific to camlistore and kept in there, just because I
wanted to keep things completely separate.
(in case you don't know, you can use
gpg [--secret-keyring /path/to/a/secring.gpg] -K
to list the keys in the secring you want to use.)
you'll get the idea anyway:

cd
mkdir .camlistore
cp the_secring_you_want_to_use.gpg .camlistore/secring.gpg
camput -secret-keyring .camlistore/identity-secring.gpg init --gpgkey XXXXXXXX

where XXXXXXXX is the gpp pubid of the key you want to use.

this will generate for you
.camlistore/config

you'll need to at least set "auth" in there. for example to
"userpass:camlistore:pass3179" if you want to match the default server
config.
also, I think there's a bug in the generation, so you should append
"bs" to the "host" in "blobGet" and "blobPut", as well as to the
"blobServer" value.
That should do it to at least get camput working.

> Thanks for submitting a fix, and for giving me a pointer on mounting; I wasn't
> using the -filenodes option before, and camput -h doesn't mention it. I'll
> start reading the source to get a better understanding of what's possible.

-filenodes is a specific option of the file mode for camput.
Just run
camput
to see the generic usage message. then,
camput file -help
will give you the help of the file mode specifically.
Although I agree camput -help should give the full usage message. will fix.

[Mathieu Lonjaret]

We have docs for that one?! :-)

Bill, as I said before, I don't use the dev-* ones myself anymore, I
just the camlistored binary directly:
with that config (for dev, not for production):
{
"listen": ":3179",
"https": false,
"auth": "userpass:camlistore:pass3179:+localhost",
"blobPath": "/home/mpl/var/camlistore/blobs",
"dbname": "camlimpl",
"identity": "F2AEB2C1",
"identitySecretRing": "/home/mpl/.camlistore/identity-secring.gpg",
"mongo": "",
"mysql": "camli@localhost:somepass",
"publish": {},
"replicateTo": [],
"s3": ""
}
(so, with mysql most of the time.)