Camlistore Roadmap (camlistore@googlegroups.com)

[Brad Fitzpatrick (Google Drive)]

I've shared an item with you.

Camlistore Roadmap

Google Drive: create, share, and keep all your stuff in one place.

[Brad Fitzpatrick (Google Drive)]

I've shared an item with you.

Camlistore Roadmap

Snapshot of the item below:

Camlistore Roadmap

Last updated 2012-10-20

Phase 1

Here’s where Camlistore is currently at as of 2012-10-20.  Let’s retroactively call it “Phase 1”.

* we have a protocol,

* we have a data/storage model,

* we have a good indexing story,

* we have a good implementation with lots of storage backends and indexing backends,

* we have lots of good tests

* we run either stand-alone or (kinda, with some bitrot) on App Engine,

* we have most of a sharing story / model, and partial implementation,

* we have a GC story, but no implementation

* we have an ugly web frontend for both owner management, and also publishing (also ugly)

… but we’re not really using it yet.

Phase 2

For the next stage of Camlistore development (let’s call it “Phase 2”) we now need to start using it ourselves, a lot.  Only our own usage will drive usability, improve performance, and find bugs.

Here’s my goal for the next phase of development.  Some people might call this a “story” or something.  But this is where we should be at the end of Phase 2:

1. All my computers are backed up regularly as static snapshots.

1. Certain file types (at least, say, image files) are also in the process available as content-based permanodes.
2. I can easily cammount (read-only) any computer’s backup at any snapshot date.

2. S3 mirroring is rock-solid.

1. sync should persist on disk where it’s at (“I’ve replicated everything before ‘sha1-f00ba4.....’) so an already-configured camlistored restart resumes syncing.

3. All the pictures I take on my phone are backed up regularly, with one permanode per photo.  This happens both

1. on-demand (“Share to Camlistore” on either a single or multiple photos, with progress bars and ongoing notifications)
2. automatically (all photos) when the phone is both power and wifi.
3. Modify existing Android app to use camput as child process (Go cross-compiled to ARM).  I have code for this.

1. The web UI has a photo-specific mode, to show me all recent photo permanodes, with

1. infinite scrolling page of thumbnails.  no “pages”
2. Bonus: box to jump to a date.
3. I can select, range-select, multi-select, unselect thumbnails, then take mass actions on them:

1. make a set (e.g. first grouping a time series into an event “All July 2012 camping photos”, then further grouping into “Good July 2012 Camping Photos”, then sharing either/both)

2. Good photo publishing mode.  Doesn’t need to be infinitely configurable yet.  One good fixed design is fine.

4. Needs to support either public URLs or secret links.
5. Must support downloading a tarball/zip of all photos.  (bonus: tar.gz/zip also includes a zip of all camlistore metadata blobs for others to import into their camlistore repos)
6. Goal: I can serve pics from http://bradfitz.com/pics/<galleryname> or /pics/<galleryname>?key=xxxx or something

Problems foreseen and other TODOs for the above:

1. Will require campput’s what-have-I-down/sha1’d caching to be better.  We were waiting on leveldb-go, but we might need a crutch in the meantime. diskkv or something?
2. EXIF library into third_party, and/or developed.  get photo date out of JPEGs.  (later: index GPS too with geospatial index.  for now extract the GPS as an attribute)
3. file chunking needs to solidify.  size histograms look too small.  we need different behavior for the beginnings and ends of files.  We need to look at file(1) and EXIF access patterns, including extracting EXIF thumbnails.

1. Goal: open “gthumb” on a remote camlistore repo with minimal bandwidth and high latency and have it be usable, with minimal transfers to get the EXIF thumbnails.
2. We should have a mode to camlistore or cammount where it injects that latency and limits the bandwidth for testing.

[TODO: add links to bugs for each item above, creating bugs where necessary]

Phase 3, 4, ….

Phase 3:

- start making it at least geek-friendly.  not the masses, but halfway technical people.

- better FUSE, including writable fuse.

- I’m using camlistore and bradfitz.com/* publishing for blog posts / “social media” stuff too, not just for photos.

- import/export picasa, twitter, g+, foursquare, etc.

- GC (per Mathieu request)

- …

Phase n:

- your non-techie friends/family can use it

- it won’t be called “Camlistore”.  it will have an end-user name.