Introducing RecordIO library: A new extremely efficient Database for AppEngine
Andrin von Rechenberg
A RecordIO is a database for (key, value) pairs for Google App Engine.
- RecordIOs can hold terabytes of data.
- Extremely efficient in writing many small entries.
- Extremely powerful because the size of entry can exceed 1MB, which means you can store bigger objects in RecordIOs than in App Engines Datastore.
What RecordIOs are good for?
In case you want to write a lot of key value pairs to datastore, RecordIOs are for you. RecordIOs use much fewer datastore operations than if you would create a single model entity for every key value pair.
Why is it better than datastore?
Cheaper and faster
Multiple key value pairs are combined and compressed into a one datastore entry which makes writing many key value pairs more efficient, especially if you can write asynchronously. In this case, writes are added to a queue and after a pre-set amount of time, all writes are applied at the same time and combined whenever possible. This minimizes datastore operations and therefor costs.
Store objects of arbitrary size
RecordIO can store objects of any size. If you insert an object that is bigger than 1MB it gets split into smaller chunks and distributed over multiple datastore entries, all automagically.
Where are RecordIOs stored?
The RecordIOs end up in datastore as sharded data. Before asynchronous writes are applied to the shards in datastore, they are queued in a normal TaskQueue.