habitat db cleanup

[Daniel Richman]

Hi all,

FYI:

As you may know, habitat (the app that parses and feeds the tracker with data) uses couchdb as its primary database. After 8 years of use, it was getting unmanageably large, we were running out of disk space, and the performance of couchdb was degrading; it was starting to choke.[1]

To fix this, a couple weeks back I emptied out the database.

 - Flights in the last two years (including flights in the future), the payload configuration documents that they reference, and any payload configuration documents added in the last month were preserved.

 - All flights ever, and all payload configuration documents ever, are preserved here: http://habitat.habhub.org/files/2020-07-habitat-db-cleanup/flights.json and here: http://habitat.habhub.org/files/2020-07-habitat-db-cleanup/payload-configuration.json; you should be able to find anything you lost in there. It is possible to restore them. You can ask for help in #habhub FreeNode IRC if you need.

 - FTR, the scripts used are here: https://gist.github.com/danielrichman/107cec7190c9111b0e8b5c7e68bcb84c

 - Actually, I renamed the old database and started a job that dumps the entire contents of the old database to a compressed file. This means that we have preserved all the payload and listener telementry docs. If you would like to recover some old data from your flight, you can grep them out of this file here: http://habitat.habhub.org/files/2020-07-habitat-db-cleanup/all-docs.json.gz, but it’s 9G, so please only download it if you need it. (This process took two weeks, hence only sending this email now.)

Thanks,

Daniel

[1] There’s a combination of effects that lead to this

 - We made a mistake: we use the sha256 of the input string as the document ID, but couchdb uses a btree to store data; thus every insertion is likely to touch a long path through the tree. Optimal use of couchdb demands that you randomly choose an ID at startup (or use the endpoint that picks one for you) and then insert successive documents with sequential IDs after that, so that the database can batch the updates together and thereby have to update fewer btree nodes per document.

 - CouchDB is architected as an append-only file. Those updated btree nodes are just written onto the end, superseding the previously written copy of that node. While we only had 31G uncompressed JSON (10G gzipped), the store was actually 800G in size, presumably mostly due to stale superseded nodes earlier in the file. It has a garbage-collection operation (“compact”), which (a) is a copy collection, and (b) extremely slow (at least a months) and (c) was just crashing on our 800G database.

 - CouchDB kinda sucks.

[Anthony Stirk]

Thanks for sort this out Daniel.

--
You received this message because you are subscribed to the Google Groups "UKHAS Tools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ukhas-tools...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ukhas-tools/CAMfJN-iADDOiSwFQgRVYn1LOjZroYrZzHrEW%3D0zCkrW0AHtsww%40mail.gmail.com.

	Anthony Stirk Managing Director T: 01274 550910 E: Anthon...@nevis.co.uk | W: http://www.nevis.co.uk
Nevis Computers is a limited company registered in England and Wales. Registered number: 3356647. Registered office: York House, 16 Wool Gate, Cottingley Business Park, Bingley, West Yorkshire, BD16 1PE. Emails may be monitored for the purposes of security and staff training.