State of the Keep: Feb 2018

[Brad Fitzpatrick]

Keepers,

It's been awhile since we posted a Perkeep status update. (Actually, we never have, because the last time we did a status update it was called Camlistore :))

Anyway, here are some notable things recently done or in progress or on the horizon:

# Rename from Camlistore to Perkeep (https://github.com/perkeep/perkeep/issues/981)

The rename is still underway. Many things have moved, but plenty remains. Notably, the mailing list hasn't been renamed, and most of the binary names (camget, camput, camlistored, etc) haven't been renamed.

But the website is nice & pretty now! See https://perkeep.org/ And we have a mascot... https://perkeep.org/keepy  Thanks, Ashley!

# Funding (https://opencollective.com/perkeep)

We now have 12 backers and 1 organization funding Perkeep at. Thank you all! We'll try to provide regular updates (& progress!) for you.

We're also doing some brainstorming on how we might get more small contributors from users and/or some larger/corporate contributors. We'll see. Nothing concrete to share yet.

The goal is still to hire a second full-time developer, in addition to Mathieu.

# Migration away from SHA-1 (https://github.com/perkeep/perkeep/issues/537)

Perkeep was always designed to change hash functions over time, which is why all blobrefs contain the hash name ("sha1-xxxxx"), but since we never actually supported anything besides SHA-1, some sha1-only assumptions had snuck into the code. Now that SHA-1 is broken, we need to move away from it. We're initially supporting SHA-224 as the new default. All new blobs will be SHA-224, and all old SHA-1 blobs will still work. We're finishing up a few corner cases now. This is a requirement before the next release. After this, adding new hash functions will be trivial, should the need inevitably arise.

# Vendor directory cleanup

Paul Lindner went on a mission to clean up our vendor/* directory, updating packages, removing unused packages, and making it all compatible and manageable cleanly with the dep tool. It's all nice and clean now. Thanks, Paul!

# Code cleanup

As part of the rename, we've taken the package renaming time as a good point to move some boring internal packages into perkeep.org/internal/*. Also, Paul fixed a bunch of "golint" stylistic things as well.

# Docs

We've started to flesh out the docs on the website more. After Will Markdown-ifying the website, we're aiming for readable docs on the web, in the code, on GitHub, and in Gerrit's git browser. The links are tricky, but we're getting there.

# Old code reviews

We did a pass over all the ancient open code reviews and either closed them, merged them, or finished them up for the original author if they were 90% of the way there already. As a result, we got some good stuff merged, including encryption changes from Filippo Valsorda, and rollsum CPU optimizations from Josh Bleecher Snyder.

# Client package simplification

The perkeep.org/pkg/client package constructors have been simplified. It had grown way too complicated. This is in prep for applications and importers and third-party programs to use Perkeep more easily.

# Google Photos Importer (https://github.com/perkeep/perkeep/issues/969, https://github.com/perkeep/perkeep/issues/896)

The Google Photos importer is in a sad state and needs love. It's a current priority. The "picasa" API importer can only see the most recent 10,000 photos or so. That's a holdover limit from the old Picasa Web Albums days, but now that Google Photos has a different data model (bunch of photos without galleries), that means the Picasa API isn't a viable way to get your photos out of Google. Unfortunately the only other API is via users manually enabling the Google Drive mirror, and then using the Google Drive API. We have that mostly working, but it needs polish. That's what we want to get done before the next release. Brad has some photos stuck in Google and wants them out.

# iOS 11 HEIC images (https://github.com/perkeep/perkeep/issues/969)

As of iOS 11, iPhones and such save images as HEIC instead of JPEG, and we download them from Google Photos as HEIC. But then we can't render the thumbnails. We need to fix that, and have started.

# Web UI

The web UI has a Map mode now, to see all your photos & check-ins & tweets on a zoomable/pannable world map.

There's also a “select all” button and support for downloading directories as zip files.

# Out-of-the-box importers (https://github.com/perkeep/perkeep/issues/1005)

 

Currently, new Perkeep users who want to use the importers (for Twitter, Foursquare Swarm, Google Photos, etc) need to set up their own OAuth2 app on each site's developer console, and then copy/paste around OAuth2 ClientID/Secrets. It's kinda a pain. We want to retain that freedom for users as an option, but want to also offer an option to use shared apps that we run on perkeep.net or something. It'd involve user instances registering with perkeep.net (authenticated by their GPG identity transparently) and setting up reverse proxies to their localhost. It'd all work without user action or config, if selected when adding an importer account. This is a current focus, for new users to experience importers more easily. We believe that importers highlight Perkeep's somewhat unique ability to handle things that aren't just files. (like tweets and check-ins).

# More importers

We need more importers. But first we want to fix the Google Photos importer and make the existing importers easier to use, and then we'll go crazy adding more.

# What else?

Surely I'm missing things. Apologies if I missed other notable changes. Feel free to reply and update others on things you're excited about (whether you've done them yourself, or others have).

- Brad & Mathieu

[Brad Fitzpatrick]

Oh, we also applied to present at Linuxfest Northwest again, but they haven't announced who's accepted yet. If approved, we'd also do a hackathon around the same time. Or we might anyway.