Debugging and authentication

[Norman Walsh]

Hi,

I worked out, by grovelling through the source, that setting CAMLI_DEBUG to true would enable the -proxy_local option by which I could try to reverse engineer the way upload works.

(Apologies if I missed the protocol documentation somewhere.)

Except if I do that, I mysteriously get 401 errors from the camlistored server.

How do I create/enable/use a user ID/password combination that will work when I'm running requests through Charles proxy?

--norm

[Mathieu Lonjaret]

On 23 December 2017 at 11:09, Norman Walsh <n...@nwalsh.com> wrote:
> Hi,
>
> I worked out, by grovelling through the source, that setting CAMLI_DEBUG to
> true would enable the -proxy_local option by which I could try to reverse
> engineer the way upload works.
>
> (Apologies if I missed the protocol documentation somewhere.)

https://perkeep.org/doc/protocol/blob-upload ?
implementations at:
pkg/client/upload.go
pkg/server/uploadhelper.go

> Except if I do that, I mysteriously get 401 errors from the camlistored
> server.
>
> How do I create/enable/use a user ID/password combination that will work
> when I'm running requests through Charles proxy?

what is Charles proxy?

> --norm
>
> --
> You received this message because you are subscribed to the Google Groups
> "Camlistore" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to camlistore+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Brad Fitzpatrick]

On Sat, Dec 23, 2017 at 10:25 AM, Mathieu Lonjaret <mathieu....@gmail.com> wrote:

On 23 December 2017 at 11:09, Norman Walsh <n...@nwalsh.com> wrote:
> Hi,
>
> I worked out, by grovelling through the source, that setting CAMLI_DEBUG to
> true would enable the -proxy_local option by which I could try to reverse
> engineer the way upload works.
>
> (Apologies if I missed the protocol documentation somewhere.)

https://perkeep.org/doc/protocol/blob-upload ?
implementations at:
pkg/client/upload.go
pkg/server/uploadhelper.go

> Except if I do that, I mysteriously get 401 errors from the camlistored
> server.
>
> How do I create/enable/use a user ID/password combination that will work
> when I'm running requests through Charles proxy?

what is Charles proxy?

It's the first hit on Google... :-) https://www.charlesproxy.com/

Norman, implementing by mimicking can get you pretty far, but be sure to actually read the protocol docs as well.

And to be extra confident in your implementation, we have tests that you can run against your server to see that it behaves correctly. I can find those later for you if you don't find them first.

[Norman Walsh]

Brad Fitzpatrick <br...@danga.com> writes:
> Norman, implementing by mimicking can get you pretty far, but be
> sure to actually read the protocol docs as well.

Absolutely. I am, however, finding the documentation difficult to understand.
For example, this page

https://perkeep.org/doc/protocol/blob-upload

Seems to be incomplete. Or at least, it doesn’t seem to describe what
camput actually does.

In my investigation, camput appears to POST the content addressable
blob then a moment later POST a blob of JSON that is apparently the
metadata for that blob. However, the metadata post is not, in any way
that I can see, distinguished as a metadata blob. And despite being
clearly a junk of JSON, it’s posted with a content-type of
application/octet-stream, so it’s inappropriate for the server to
attempt to parse it.

> And to be extra confident in your implementation, we have tests that
> you can run against your server to see that it behaves correctly. I
> can find those later for you if you don't find them first.

Thanks. Absent some documentation or explanation for what the protocol
is actually doing, I’m not sure how much more progress I can make.

Be seeing you,
norm

--
Norman Walsh <n...@nwalsh.com> | What is familiar is what we are used
http://nwalsh.com/ | to; and what we are used to is most
| difficult to 'Know'—that is, to see as
| a problem; that is, to see as strange,
| as distant, as 'outside us'.--Nietzsche

[Brad Fitzpatrick]

On Tue, Dec 26, 2017 at 2:45 AM, Norman Walsh <n...@nwalsh.com> wrote:

Brad Fitzpatrick <br...@danga.com> writes:
> Norman, implementing by mimicking can get you pretty far, but be
> sure to actually read the protocol docs as well.

Absolutely. I am, however, finding the documentation difficult to understand.
For example, this page

  https://perkeep.org/doc/protocol/blob-upload

Seems to be incomplete. Or at least, it doesn’t seem to describe what
camput actually does. 

In my investigation, camput appears to POST the content addressable
blob then a moment later POST a blob of JSON that is apparently the
metadata for that blob. However, the metadata post is not, in any way
that I can see, distinguished as a metadata blob. And despite being
clearly a junk of JSON, it’s posted with a content-type of
application/octet-stream, so it’s inappropriate for the server to
attempt to parse it.

A blob server just stores blobs. It should not try to interpret blobs. In a blob, there is no difference between data and metadata, and there are not even content types in blobs, hence  application/octet-stream for everything.

The blob-upload protocol docs don't get into that, because that all happens at layers above blob storage.

When you said you waited to write a "backend", I assumed you meant a blob storage backend, but perhaps you meant something else. Which part do you want to write, and for what reasons? Maybe we can point you in the right direction.

[Norman Walsh]

Brad Fitzpatrick <br...@danga.com> writes:
> When you said you waited to write a "backend", I assumed you meant a
> blob storage backend, but perhaps you meant something else. Which
> part do you want to write, and for what reasons? Maybe we can point
> you in the right direction.

Hmm. Sorry, I guess I don’t really understand how all the pieces fit together.
I was imagining that the backend was responsible for storing, indexing, and
querying the data. Perhaps even generating some of the /ui/.

Be seeing you,
norm

--

Norman Walsh <n...@nwalsh.com> | The Future is something which everyone
http://nwalsh.com/ | reaches at the rate of sixty minutes an
| hour, whatever he does, whoever he
| is.--C. S. Lewis

[Brad Fitzpatrick]

On Wed, Dec 27, 2017 at 12:21 AM, Norman Walsh <n...@nwalsh.com> wrote:

Brad Fitzpatrick <br...@danga.com> writes:
> When you said you waited to write a "backend", I assumed you meant a
> blob storage backend, but perhaps you meant something else. Which
> part do you want to write, and for what reasons? Maybe we can point
> you in the right direction.

Hmm. Sorry, I guess I don’t really understand how all the pieces fit together.
I was imagining that the backend was responsible for storing, indexing, and
querying the data. Perhaps even generating some of the /ui/.

I'll consider this confusion a TODO to update our docs.

We have https://perkeep.org/doc/arch but it's missing some bubbles. We should add apps, importers, and /ui/, all of which are just different types of clients of the search interface (which uses the indexes, shown) + blob storage interface.

There are basically two storage abstractions:

-- storing content-addressable small-ish blobs (16MB max, currently). this is the blob upload protocol we pointed you at before.

-- the index storage (anything that can implemented a sorted key/value --- https://perkeep.org/pkg/sorted#KeyValue). But this is currently Go only, and we already basically have Go implementations of every database.

Deciding what & how to index things is currently not an abstraction. That's all in Go. But one could in theory implement the REST interface to the search interface, but that's a moving target and ever growing, and you'd have to then do all your own indexing. You'd be forever playing catch-up, so we focus on the one Go implementation.

I had assumed you were only interested in a new blob storage backend.

[Norman Walsh]

Brad Fitzpatrick <br...@danga.com> writes:
> Deciding what & how to index things is currently not an abstraction.
> That's all in Go. But one could in theory implement the REST
> interface to the search interface, but that's a moving target and
> ever growing, and you'd have to then do all your own indexing. You'd
> be forever playing catch-up, so we focus on the one Go
> implementation.
>
> I had assumed you were only interested in a new blob storage backend.

Ah well, nevermind then. I was going to implement a backend in
MarkLogic Server but I guess there’s no point.

Be seeing you,
norm

--

Norman Walsh <n...@nwalsh.com> | There has never been a perfect
http://nwalsh.com/ | government, because men have passions;
| and if they did not have passions,
| there would be no need for
| government.--Voltaire