Re: gogallery and Camlistore

[Mathieu Lonjaret]

Ok, interestingly on another machine (and probably a pretty similar
camlistore version, can't compare right now), all the ui stuff seems
to work. Same OS (ubuntu 64 bits), both recent go pulls. I'll
investigate that later.
Do you have any pointer/instruction on where you'd like me to start or
should I just start reading and poking around in the code?

On Tue, Jun 7, 2011 at 11:37 PM, Brad Fitzpatrick <brad...@golang.org> wrote:
> We should move this thread to the Camlistore mailing list.
> But in a nutshell: a lot of the UI stuff is very new (last few hours - last
> 2 weeks) & still incomplete.  Let's talk about what you need next on the
> list.
> On Tue, Jun 7, 2011 at 2:21 PM, Mathieu Lonjaret
> <mathieu....@gmail.com> wrote:
>>
>> Ok, it took me a while to find out that I had to change the hardcoded
>> pass for the db in the dev-server script as well, not only in the json
>> config file.
>> from the web ui, it seems like I can upload files, but the "New"
>> button to create an item or a collection doesn't seem to do anything.
>> And after some poking around everywhere in the ui, I have that in the
>> blobs:
>>
>> +-----------------------------------------------+---------+----------------------------------+
>> | blobref                                       | size    | type
>>                      |
>>
>> +-----------------------------------------------+---------+----------------------------------+
>> | sha1-ad87ca5c78bd0ce1195c46f7c98e6025abbaf007 |     448 |
>> text/x-openpgp-public-key        |
>> | sha1-f8f964331808fa428de001cf47f107fa5970adc0 |     218 |
>> application/json; camliType=file |
>> | sha1-04f36957b2eb157513d38498a2b98808da9b9bc1 |  164375 | image/jpeg
>>                      |
>> | sha1-496f546f8c6a1444c0944cecc942b5dc4c74317a | 1048576 | image/jpeg
>>                      |
>> | sha1-8df06bb9f431334b04a1739a2144480153997aae |     317 |
>> application/json; camliType=file |
>> | sha1-be07712f747b2a7274eb3d80c3ba9b9754d1eb05 |  517529 | NULL
>>                      |
>> | sha1-a1e5159e74514d07a24fee08b2e6d6d0c759f446 |    1843 | image/jpeg
>>                      |
>> | sha1-8c52a96b8e0db4a3469ab7e4291e27787e8a1d0f |   11372 | NULL
>>                      |
>> | sha1-ade916e311ddb65b8c21a768afb99e72c3df003c |    7399 | NULL
>>                      |
>> | sha1-144be469e6ed46d146b7081bb12045296d87036b |    1562 | NULL
>>                      |
>> | sha1-a9b64a871368a64b763187788caabd8b11193d29 |    3875 | NULL
>>                      |
>> | sha1-3864f707027d138449025ef8704344ce5571f2a3 |    1150 | NULL
>>                      |
>> | sha1-37ed9d5b97dcf99f76200495baf58fb86516881c |   10487 | NULL
>>                      |
>> | sha1-cf4f9da7516f669502f2c656e6698c7ff152e393 |    4366 | NULL
>>                      |
>> | sha1-99683a5ea6dde7c6e9ce82cde5c3ab68af2d279f |    5099 | NULL
>>                      |
>> | sha1-3045a3b700382f752866eb82287ad332c93c4b09 |     575 |
>> application/json; camliType=file |
>> | sha1-8b3d3b07901dddb9ec44ccfaa86adc9bef40c421 |   16553 | NULL
>>                      |
>> | sha1-54cc7761804bcf66de77bc5af4f080d480ea5757 |     281 |
>> application/json; camliType=file |
>> | sha1-358c9411a38208eb9be3a3fbf7f9511b9ae62c9b |     630 |
>> application/json; camliType=file |
>>
>> +-----------------------------------------------+---------+----------------------------------+
>>
>> am I doing something wrong or is that how it is supposed to be at the
>> moment?
>
>

[Brad Fitzpatrick]

For now I'd focus on the permanode.{html,js} page in the UI.

It should be possible to drag & drop or (multi-)upload photos into a gallery, creating new permanodes for each, and adding new "child" attributes for each child photo in the gallery.  I'm working on some of that right now, so perhaps you should focus on the image-specific stuff.  Note the existing download handler that reconstructs a byte stream from a massively cut up "file" schema blob.  You should probably make something like that for resized thumbnails taking maxw and maxh query parameters.  And ideally without imagemagick, since we want this to run on Linux, Windows, App Engine, Mac, etc.

That work for a starter project?

[Brad Fitzpatrick]

What time zone are you in?  Perhaps we could hack on this together sometime over IM in the next day or so?

[Mathieu Lonjaret]

On Wed, Jun 8, 2011 at 7:39 PM, Brad Fitzpatrick <br...@danga.com> wrote:
> What time zone are you in?  Perhaps we could hack on this together sometime
> over IM in the next day or so?

I'm in GMT+2 atm. I'll have a couple hours free in an hour or so.
I should have more time tomorrow, starting at around 19:00 GMT (and
sporadically before).

> On Wed, Jun 8, 2011 at 10:38 AM, Brad Fitzpatrick <br...@danga.com> wrote:
>>
>> For now I'd focus on the permanode.{html,js} page in the UI.
>> It should be possible to drag & drop or (multi-)upload photos into a
>> gallery, creating new permanodes for each, and adding new "child" attributes
>> for each child photo in the gallery.  I'm working on some of that right now,
>> so perhaps you should focus on the image-specific stuff.  Note the existing
>> download handler that reconstructs a byte stream from a massively cut up
>> "file" schema blob.  You should probably make something like that for
>> resized thumbnails taking maxw and maxh query parameters.  And ideally
>> without imagemagick, since we want this to run on Linux, Windows, App
>> Engine, Mac, etc.
>> That work for a starter project?

sounds fun. However I think we're not gonna go far without imagemagick
or an equivalent as afaik the standard img lib is quite far from being
there yet (they can't decode just any jpeg, I retried a few days ago),
and there's no project that does image rescaling natively afaik. I had
started one but I had stalled to work on more pressing things. I guess
maybe it's time I restarted on that.

[Brad Fitzpatrick]

On Wed, Jun 8, 2011 at 1:41 PM, Mathieu Lonjaret <mathieu....@gmail.com> wrote:

On Wed, Jun 8, 2011 at 7:39 PM, Brad Fitzpatrick <br...@danga.com> wrote:
> What time zone are you in?  Perhaps we could hack on this together sometime
> over IM in the next day or so?

I'm in GMT+2 atm. I'll have a couple hours free in an hour or so.
I should have more time tomorrow, starting at around 19:00 GMT (and
sporadically before).

Feel free to bug me on xmpp whenever (bradfitz gmail.com).  If I'm red that just means I may not be around the computer, but I should be around at 19:00 GMT tomorrow and I'll plan on working then.  Same goes for anybody else.  Perhaps we can group chat or IRC if that fails.

 

> On Wed, Jun 8, 2011 at 10:38 AM, Brad Fitzpatrick <br...@danga.com> wrote:
>>
>> For now I'd focus on the permanode.{html,js} page in the UI.
>> It should be possible to drag & drop or (multi-)upload photos into a
>> gallery, creating new permanodes for each, and adding new "child" attributes
>> for each child photo in the gallery.  I'm working on some of that right now,
>> so perhaps you should focus on the image-specific stuff.  Note the existing
>> download handler that reconstructs a byte stream from a massively cut up
>> "file" schema blob.  You should probably make something like that for
>> resized thumbnails taking maxw and maxh query parameters.  And ideally
>> without imagemagick, since we want this to run on Linux, Windows, App
>> Engine, Mac, etc.
>> That work for a starter project?

sounds fun. However I think we're not gonna go far without imagemagick
or an equivalent as afaik the standard img lib is quite far from being
there yet (they can't decode just any jpeg, I retried a few days ago),

Oh?  It should be able to.  If you find one it can't, that's a bug.  It'll be fixed.  A bunch of work on the images packages and a compression bug affecting images was fixed in the past few weeks.  

 

and there's no project that does image rescaling natively afaik. I had
started one but I had stalled to work on more pressing things. I guess
maybe it's time I restarted on that.

I thought somebody did it already.  I'll look around.  Using imagemagick just isn't interesting enough, severely limiting where we can run.

[Mathieu Lonjaret]

On Thu, Jun 9, 2011 at 1:13 AM, Brad Fitzpatrick <br...@danga.com> wrote:
> On Wed, Jun 8, 2011 at 1:41 PM, Mathieu Lonjaret
> <mathieu....@gmail.com> wrote:
>>
>> On Wed, Jun 8, 2011 at 7:39 PM, Brad Fitzpatrick <br...@danga.com> wrote:
>> > What time zone are you in?  Perhaps we could hack on this together
>> > sometime
>> > over IM in the next day or so?
>>
>> I'm in GMT+2 atm. I'll have a couple hours free in an hour or so.
>> I should have more time tomorrow, starting at around 19:00 GMT (and
>> sporadically before).
>
> Feel free to bug me on xmpp whenever (bradfitz gmail.com).  If I'm red that
> just means I may not be around the computer, but I should be around at 19:00
> GMT tomorrow and I'll plan on working then.  Same goes for anybody else.
>  Perhaps we can group chat or IRC if that fails.

ok, thanks.

>> > On Wed, Jun 8, 2011 at 10:38 AM, Brad Fitzpatrick <br...@danga.com>
>> > wrote:
>> >>
>> >> For now I'd focus on the permanode.{html,js} page in the UI.
>> >> It should be possible to drag & drop or (multi-)upload photos into a
>> >> gallery, creating new permanodes for each, and adding new "child"
>> >> attributes
>> >> for each child photo in the gallery.  I'm working on some of that right
>> >> now,
>> >> so perhaps you should focus on the image-specific stuff.  Note the
>> >> existing
>> >> download handler that reconstructs a byte stream from a massively cut
>> >> up
>> >> "file" schema blob.  You should probably make something like that for
>> >> resized thumbnails taking maxw and maxh query parameters.  And ideally
>> >> without imagemagick, since we want this to run on Linux, Windows, App
>> >> Engine, Mac, etc.
>> >> That work for a starter project?
>>
>> sounds fun. However I think we're not gonna go far without imagemagick
>> or an equivalent as afaik the standard img lib is quite far from being
>> there yet (they can't decode just any jpeg, I retried a few days ago),
>
> Oh?  It should be able to.  If you find one it can't, that's a bug.  It'll
> be fixed.  A bunch of work on the images packages and a compression bug
> affecting images was fixed in the past few weeks.

well, maybe I was unlucky, but with the first image I had tried (just
wanted to decode it to get its width and height), I hit one of those
two errors (can't remember which one exactly).

UnsupportedError("luma downsample ratio")
UnsupportedError("chroma downsample ratio")

(src/pkg/image/jpeg/reader.go:162)

>> and there's no project that does image rescaling natively afaik. I had
>> started one but I had stalled to work on more pressing things. I guess
>> maybe it's time I restarted on that.
>
> I thought somebody did it already.  I'll look around.  Using imagemagick
> just isn't interesting enough, severely limiting where we can run.

if you don't find anything, I had some very basic stuff working here I think:
https://bitbucket.org/mpl/scale/src/eed2a83bd9a5/scale.go
ah, and I've just found the thread where some guy announced he had
made some bindings for GD. And Nigel Tao says there's some resizing
going on in moustachio:
http://code.google.com/p/appengine-go/source/browse/example/moustachio/resize/resize.go

Since I was on my home machine where some of the ui stuff isn't
working (even after fresh pull from both go and camlistore) -
specifically the button to create a permanode doesn't seem to do
anything - I was trying to figure out what's wrong before I can go on.
So far, I've followed it it as far as into camliSigDiscovery
(camli.js) where I don't see any problem, but I don't know much
javascript so I'm not sure.

gotta crash to bed, ttyl.

[Mathieu Lonjaret]

Hi,

here's the first draft for the thumbnail handler:
(I wasn't comfortable directly committing directly to camlistore git
as this is still a wip; it's not tested but it compiles).
https://github.com/mpl/camlistuff

The two main questions I have:
1) Do I store the newly created thumbnail as a new permanode in camli
(I think you mentionned that, not sure anymore), so we can reuse it
later? or do we just serve it and forget about it?
2) how do we go about the resize "package" that I created but that is
directly extracted from moustachio? Do we keep it inside? Do I ask the
moustachio guys if they think it would be a good idea that I make an
official external package out of it so they (and us) can import and
reuse it as is? or do I extract the minimum that we actually need and
have it as a new func somewhere?

Cheers,
Mathieu

[Brad Fitzpatrick]

On Fri, Jun 10, 2011 at 2:37 AM, Mathieu Lonjaret <mathieu....@gmail.com> wrote:

Hi,

here's the first draft for the thumbnail handler:
(I wasn't comfortable directly committing directly to camlistore git
as this is still a wip; it's not tested but it compiles).
https://github.com/mpl/camlistuff

Oh, I thought you'd just clone + patch the whole repo, not just a couple files.

Anyway, feedback:

Don't include the query parameters in your pattern:  just match on URL.Path, not URL.RawPath

var thumbnailPattern = regexp.MustCompile(`^thumbnail/([^/]+)(/.*)?\?mw=([0-9]+)\?my=([0-9]+)$`)

 

Where do you call serveThumbnail?  It's defined, but not dispatched to.

//TODO: Do we try to keep things proportional, or do we trust the input params?

Yes.  Keep proportional.

The two main questions I have:
1) Do I store the newly created thumbnail as a new permanode in camli
(I think you mentionned that, not sure anymore), so we can reuse it
later? or do we just serve it and forget about it?

No creation.  Just a read-only operation.  We can add caching (memory, disk) later.

 

2) how do we go about the resize "package" that I created but that is
directly extracted from moustachio? Do we keep it inside? Do I ask the
moustachio guys if they think it would be a good idea that I make an
official external package out of it so they (and us) can import and
reuse it as is? or do I extract the minimum that we actually need and
have it as a new func somewhere?

For now, until it's part of the standard Go library, just put it in lib/go/camli/misc/resize (make a new directory).

The stuff in camli/misc is meant to be moved upstream or elsewhere later.  If it goes to Go, we'll have it.  If it goes to github or somewhere else, then Camlistore sucks it into lib/go/camli/third_party/ to ensure no dependency management for users and there's always a stable snapshot to build from, with all deps.

When you make a new directory (== new package) in Camlistore, be sure to register it with the "build.pl" file in the top-level directory.  No Makefiles are used.  (they're auto-generated for now)

[Mathieu Lonjaret]

Alright, thanks.

to answer your question:

> Where do you call serveThumbnail?  It's defined, but not dispatched to.

Yep, haven't done that yet. will do right after I address your other comments.

[Mathieu Lonjaret]

Hi,

I reworked all that on the fork I made on github.
I tried testing from the ui (hacked a bit of blobinfo*) but the
results were erratic since I don't grok well the js yet. However,
things went well when trying a few cases directly from the url bar.
(after having uploaded a pic and directly using its sha-1 in the url).
I also got another jpeg decoding error (the decoder can't handle yet
"progressive mode", whatever that is) with one of the pics I tried.

https://github.com/mpl/camlistore

Do you want me to make a pull request from github?
Also, how do you keep in sync your repo on github and the one reached
through gitosis on camlistore.org ?

[Brad Fitzpatrick]

Mathieu,

Looks mostly good but I do have a few comments before I merge.  Can you squash your two commits into one and use "git cl upload" to upload the CL for review to codereview.appspot.com?

[Mathieu Lonjaret]

Done, as http://codereview.appspot.com/4585051/

I used misc/codereview/git-cl, I hope that's what you meant.

[Mathieu Lonjaret]

Hi,

As a next step and a sort of proof of concept, I was thinking I could
try and hack on blobinfo.*, so that instead of the filename (as the
text for the link to download the pic), the thumbnail for the pic
would be displayed. How does that sound?

Unless you already have another idea of a direction for me to follow
of course...

Mathieu

[Brad Fitzpatrick]

Sounds perfect.  I assumed that'd be coming up and worked on the indexer last weekend so it now knows the sniffed mime type of file schema blobs' contents.  All you should have to do is check to see if it starts with "image/" to wire it up to your new thumbnail handler.  I'd still keep the filename in the link, just in case the thumbnailer can't handle it, but also show the thumbnail if appropriate.

[Mathieu Lonjaret]

So I googled a bit on how to get binary data (the image) from the
response of an
XMLHttpRequest in javascript and it seems like it's a bit hackish and
depends on the browser, see for example:
http://emilsblog.lerch.org/2009/07/javascript-hacks-using-xhr-to-load.html
So maybe things have improved since that post and there's a clean way
to do that nowadays, but if not, I'd rather make another handler on
the go side (if possible on top of the first one) that would write the
thumbnail to a temp file and for example send back the location of the
file in the responseText. Would that be ok?

[Mathieu Lonjaret]

Also it seems like a nice way to do that is with xhr2:
http://www.html5rocks.com/en/tutorials/file/xhr2/, so maybe that could
be the primary way and if the browser doesn't support it, resort back
to using the new handler I've just mentionned (or the old hackish
way)?

[Brad Fitzpatrick]

I'm confused.  You already wrote a handler to return the resized JPEG bytes.  Why can't you just do:

var img = document.createElement("img");

img.src = disco.uiRoot + "/thumbnail/" + blobref + "?mw=100&my=100";

It seems simple.

[Mathieu Lonjaret]

Right.
I had managed to convince myself into thinking that this wouldn't work
because the address for the src of the pic was dynamic (and thus that
I needed to prefetch it beforehand).

Thanks and sorry about the noise.

[Brad Fitzpatrick]

Phew.  I was concerned there was something I was overlooking.

I look forward to the thumbnails!  bslatkin, derat and I are hacking on Camlistore tonight, starting in about an hour, so we'll be online and hacking tonight if others want to join.

[Mathieu Lonjaret]

Hi,

I'm not sure what's going on with git/gerrit but ./misc/review is
bugging me atm. I do have have the commit-msg hook (it worked fine the
last time), and yet I get this error:
mpl@yggdrasil:~/camlistore$ ./misc/review
Counting objects: 77, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (44/44), done.
Writing objects: 100% (44/44), 5.23 KiB, done.
Total 44 (delta 23), reused 0 (delta 0)
remote: Resolving deltas: 13% (3/23)
remote: ERROR: missing Change-Id in commit message
remote: Suggestion for commit message:
remote: thumbnail support
remote:
remote: Change-Id: If7a96adc42d3dc6088b5780a8e319d37ce6fadc4
To ssh://m...@camlistore.org:29418/camlistore
! [remote rejected] HEAD -> refs/for/master (missing Change-Id in
commit message)

and I do have that in top of my git log:

commit 03604e4c3011bcb32fa4e97a53d0ae865d1d0216
Author: mpl <mathieu....@gmail.com>
Date: Fri Jun 17 13:35:32 2011 +0200

use thumbnail in blobinfo

Change-Id: I48722b88ec4f2c80b689034df1c0e3fd5132b36c

Since I gotta run and I'm not sure how much connectivity I'll have in
the next couple of days, here's the blobinfo.js attached.

[Brad Fitzpatrick]

This happened to Brett earlier.  Are you in the middle of a merge?  After you merge you should "git rebase -i origin/master" and squash your changes down to one commit before you upload.  Make sure "git status" doesn't show pending commits.

But I'll look at getting this merged regardless. 

[Martin Atkins]

On 06/17/2011 07:24 AM, Brad Fitzpatrick wrote:
> This happened to Brett earlier. Are you in the middle of a merge?
> After you merge you should "git rebase -i origin/master" and squash
> your changes down to one commit before you upload. Make sure "git
> status" doesn't show pending commits.
>
> But I'll look at getting this merged regardless.
>

Anecdote that may help: we tried to switch on the "Force Change-Id" on
my company's internal gerrit when we were already deep in git history,
and found that this didn't really work because when you push to gerrit
it often, for whatever reason, inspects older commits you depend on as
well as your current commit, and the older commits don't have Change-Id
on them, so it fails.

I don't know if you can wait for a few months after everyone has used
Change-Id exclusively and then turn it on, but certainly in my
experience just switching it on suddenly as soon as you start using
Change-Id doesn't seem to work right.

Maybe someone who knows more about the internals of gerrit could explain
why it behaves this way; I can only tell you what I observed from the
outside. :)

[Mathieu Lonjaret]

Thanks for the merge. Glad to see you changed "my" into "mh" for the query too.

[Mathieu Lonjaret]

Hey,

I should have some more time tomorrow evening/night GMT-ish. Shall we
discuss the next steps?
As far as I can see from the ui it seems like changing the type (to
collection/gallery for example) doesn't do anything atm. Shall I try
and address that? I have no idea (yet) what's already done to get that
working though.

Mathieu

[Mathieu Lonjaret]

Hi Brad,

here are the changes for searching the permanodes by tag, as we discussed:
https://github.com/mpl/camlistore/commit/ea22b2fbf45ab797701125e56ee5f5c6d88abee6

at first I had created a new table with an index as you had advised,
but when I saw your latest changes I figured I could directly reuse
signerattrvalue since it seems suitable enough. I'm probably missing a
performance issue with that though. If that's the case, couldn't we
just add another index to signerattrvalue instead of creating a new
table? I need to read some more about sql indexes...

also, the silly changes to the .js/.html are not what I'm proposing of
course, they're just a means of testing the go stuff.

Cheers,
Mathieu

[Brad Fitzpatrick]

Cool.  Want to push that to Gerrit?  (run ./misc/review)  Gerrit has key bindings and I can comment a lot faster than all the clicking required on Github. And Gerrit lets me merge it server-side too.

[Mathieu Lonjaret]

Done. I wasn't sure whether you wanted that discussed some more before
pushing to gerrit, hence why github.
Glad to see the push went smoothly with gerrit this time :)