overview-local stores files in a new place now

[Adam Hooper]

TL;DR If you use overview-local, now is a good time to ./backup.

In other words:

./stop

./backup because-adamhooper-said-so.tar.gz

./update

./start

What's causing the change?

We're moving database files into named Docker volumes.

Why use named Docker volumes?

This will make it harder to lose Overview's data by mistake.

Here's the technical problem.

overview-local uses Docker. Docker stores data in volumes: directories on your hard drive (on Linux) or in Docker's virtual-machine image (on Mac and Windows). Historically, these volumes were magically created when we created containers. (A container is a running program.) If the container disappears, nothing points to the volume: you can't find your data.

Historically, to preserve the volume, we had to preserve its container. We'd create an empty container per volume. You'd run it and it would ... stop. This was standard practice for Docker users.

Here's the problem: stopped Docker containers can clutter a system. Many Docker users run a command every once in a while that deletes all stopped containers. That deletes data-volume containers. And that deletes your data.

All that to say: in the past, a routine Docker maintenance command -- included in Docker's online documentation -- can hide or delete your Overview database.

Docker developers built a solution: named volumes. Routine Docker commands won't delete named volumes. And named volumes are easy to find on disk.

Why didn't overview-local use named volumes earlier?

Named volumes are a relatively new Docker feature.

Why don't we switch to named volumes later?

We've helped several users troubleshoot data-volume container problems. We conclude that users will continue to have trouble -- misplaced files and possible data loss -- as long as we use data-volume containers.

We must switch to named volumes to protect your data over the long term. The sooner we switch, the better.

What's the risk?

The next time you run overview-local's ./start, it will move your files from unnamed volumes to named volumes.

We designed our file-moving code to recover from errors. We've tested extensively. But we're only human.

If our code has a bug, it could affect your files. Most likely, Overview won't start and we can work with you to recover your data. (Email overview-users@groups.google.com.) But one wrong command during troubleshooting can be disastrous.

Why "move files" instead of "copy files?" Isn't copying safer?

Some of our users don't have enough disk space to hold two copies of their Overview data. Copy won't work for them.

Instead of copying, you can choose to back up manually. You can store the backup on a disk with space. Or you can skip the backup entirely.

Enjoy life,

Adam

[Porter Hall]

Dang, I really regret not backing up before trying this.

I ran my start script (installed on OS X) and got an error. I stopped the container and restarted, getting the same docker-compose note and error 1 fail:

Starting "default"...
(default) Check network to re-create if needed...
(default) Waiting for an IP...
Machine "default" was started.
Waiting for SSH to be available...
Detecting the provisioner...
Started machines may have new IP addresses. You may need to re-run the `docker-machine env` command.
Updating overview-local repository...
Launching database...
Builds, (re)creates, starts, and attaches to containers for a service.

Unless they are already running, this command also starts any linked services.

The `docker-compose up` command aggregates the output of each container. When
the command exits, all containers are stopped. Running `docker-compose up -d`
starts the containers in the background and leaves them running.

If there are existing containers for a service, and the service's configuration
or image was changed after the container's creation, `docker-compose up` picks
up the changes by stopping and recreating the containers (preserving mounted
volumes). To prevent Compose from picking up changes, use the `--no-recreate`
flag.

If you want to force Compose to stop and recreate all containers, use the
`--force-recreate` flag.

Usage: up [options] [SERVICE...]

Options:
    -d                         Detached mode: Run containers in the background,
                               print new container names.
                               Incompatible with --abort-on-container-exit.
    --no-color                 Produce monochrome output.
    --no-deps                  Don't start linked services.
    --force-recreate           Recreate containers even if their configuration
                               and image haven't changed.
                               Incompatible with --no-recreate.
    --no-recreate              If containers already exist, don't recreate them.
                               Incompatible with --force-recreate.
    --no-build                 Don't build an image, even if it's missing
    --abort-on-container-exit  Stops all containers if any container was stopped.
                               Incompatible with -d.
    -t, --timeout TIMEOUT      Use this timeout in seconds for container shutdown
                               when attached or when containers are already
                               running. (default: 10)
Command failed with error 1

Also these errors when I stop:

ERROR: Version in "/Users/myUser/overview-local/config/overview.yml" is unsupported. Either specify a version of "2" (or "2.0") and place your service definitions under the `services` key, or omit the `version` key and place your service definitions at the root of the file to use version 1.
For more on the Compose file format versions, see https://docs.docker.com/compose/compose-file/
ERROR: Version in "/Users/myUser/overview-local/config/overview.yml" is unsupported. Either specify a version of "2" (or "2.0") and place your service definitions under the `services` key, or omit the `version` key and place your service definitions at the root of the file to use version 1.
For more on the Compose file format versions, see https://docs.docker.com/compose/compose-file/
Command failed with error 1
Stopping "default"...
Machine "default" was stopped.

[Adam Hooper]

Hi Porter,

First off: your data is safe.

Beyond that: sorry for inconveniencing you! These errors are due to changes in Docker for Mac (and Docker for Windows). The long and short is: if you're using an old version of Docker for Mac or Docker for Windows, yes, you'll need to back up.

I ignored a glaring problem in our backup/restore strategy for Mac and Windows machines. I will fix this soon.

Until I say otherwise on this thread, please don't upgrade Docker yet. The fundamental problem is: the latest overview-local depends on the latest version of Docker; and Docker for Mac and Docker for Windows will hide all your data when you upgrade.

Again: your data is safe. I just need to devise new instructions for bringing it to the latest version of Docker.

If you want to use Overview in the meantime -- with all your data intact -- open a console and try this:

(cd overview-local && git fetch && git checkout ac1fe)

overview-local/start-after-git-pull

This will put you in a "detached" state. To return to normal -- which, until I fix things, will continue to cause errors for you -- run this: (cd overview-local && git checkout master)

Again, many apologies. We adopted Docker when it was very young. We're relaying its growing pains to you.

Enjoy life,

Adam

[Adam Hooper]

For those on Mac and Windows who want to back up before upgrading, my initial suggestion -- running overview-local/backup -- was misplaced. Our old backup script didn't work that way on Mac and Windows.

How to back up to a file on Mac and Windows

On Windows and Mac, we used to recommend using VirtualBox machine snapshots. That's a problem because newer versions of Docker on Mac and Windows don't always use VirtualBox. Here's a safer backup strategy: save to a file.

I suggest you open your terminal and type this:

docker run --volumes-from config_overview-blob-storage --volumes-from config_overview-database-data --volumes-from config_overview-searchindex-data --rm library/ubuntu:17.10 tar -czp -C / var/lib/postgresql/data var/lib/overview/searchindex var/lib/overview/blob-storage > ~/overview-local/backup.tar.gz

This should create a file called "backup.tar.gz". Run docker run --rm -i busybox tar ztf < ~/overview-local/backup.tar.gz to see a listing of its contents. If that listing includes "blob-storage" and "postgresql", all is well. If it doesn't, stop and email me. I haven't run this command on your machine, so I don't know for sure it will work.

After this works (and not if it doesn't), Upgrade Docker if you aren't on the most up-to-date version already. Install either "Docker for Windows" or "Docker for Mac", depending on your operating system. If you're using "Docker Toolbox", that's way too old.

Finally, Upgrade Overview using overview-local/update. This should Just Work. Your data should migrate over just fine. The "backup.tar.gz" file is a great way to recover in case anything untoward happens.

Enjoy life,

Adam

[Adam Hooper]

Whoops. Take two:

docker run --volumes-from overview-blob-storage --volumes-from overview-database-data --volumes-from overview-searchindex-data --rm library/ubuntu:17.10 tar -czp -C / var/lib/postgresql/data var/lib/overview/searchindex var/lib/overview/blob-storage > ~/overview-local/backup.tar.gz