LMFDB up and running on postgresql!
27 views
Skip to first unread message
Andrew Sutherland
Jul 25, 2018, 4:13:16 PM7/25/18
to lmdb
As some of you may know, we have been working for the past several months on migrating the data storage engine for the LMFDB from Mongo DB to Postgresql. We are happy to report that this migration is now essentially complete, and is up and running at
We invite everyone to checkout this new incarnation of the LMFDB which should be functionally identical to the LMFDB you know and love, except that many operations will now be faster, and several queries that previously resulted in timeout errors can now be completed in a few seconds. To see some examples of this, you can check out the "before" and "after" links at
Our hope is that you will take the opportunity to try out the new system and report any issues you may find at
Assuming all goes well, our plan is to switch www.lmfdb.org to point to a new cloud-based postgresql server in the next 2-3 weeks. This change should be essentially invisible to all current users of the LMFDB (we will of course keep the existing mongo DB system running in parallel for some time as a backup). We will let everyone now once we have fixed an exact date for the switch.
Following the changeover of www.lmfdb.org, we will also switch beta.lmfdb.org to point to a new postgresql-based development system running on new hardware. This should also be transparent to users of beta.lmfdb.org, but for developers there will be changes to how new data is added to the LMFDB (documentation and details on this will follow). A FAQ for developers that explains the ins and outs of writing LMFDB code based on postgresql will be made available before the changeover (but the good news is the changes are minimal).
For developers who are interested in the code changes involved in this migration, you can check out a complete postgresql-based branch of the LMFDB code at
At the time of switchover we will merge the postgres branch into the master branch of the LMFDB. In order to make this process as smooth and painless as possible, we would like to discourage the submission of non-trivial PRs between now and then (we have kept the postgres branch up to date with the master branch during the development prosses, but we would prefer to have a quiet period during which no new code is merged into the postgres branch before the changeover). After the changeover, PRs will be merged into the new master branch.
This work has been in the planning stages for many months now, and we are grateful for all the hard work put in by Edgar Costa and (especially) David Roe to make this possible. The new postgresql platform will not only address many outstanding performance issues, it should allow us to provide much more extensive query facilities to users of the LMFDB in the future.
John Cremona and Andrew Sutherland
Andrew Sutherland
Aug 10, 2018, 11:40:16 AM8/10/18
to lmdb
The code changes related to the conversion to postgres have now been merged into the master branch and pushed to two new branches named "dev" and "web" that will takeover the roles of beta and prod going forward (the plan is to keep "beta" and "prod" as snapshots of the last mongo-db version).
There are two new cloud-based webservers http://dev.lmfdb.xzy and http://web.lmfdb.xyz that are now running the code in the dev and web branches respectively; the webservers at http://beta.lmfdb.org and http://www.lmfdb.org are still running on the mongo-db branches beta and prod. The plan is to switch the domain names beta.lmfdb.org and www.lmfdb.org to point to the new servers running the postgres code some time in the next two weeks. We will keep the mongo-db servers up and running for some period of time after the switch that can be accessed via the domain names beta.lmfdb.xyz and prod.lmfdb.xyz.
With this change, all new PRs will need to be merged into the postgres-based master. If you are a current LMFDB developer, the only change you need to make to your development environment is to execute the command
sage -pip install psycopg2-binary
on your development machine. This will install the python libraries necessary to talk to postgres. You should do this before you sync your local repository with the new upstream master.
If you are a new LMFDB developer (or setting up a new LMFDB development environment) this change will be transparent to you, the psycopg2-binary packaage has been added to requirements.txt and the "getting started" instructions at https://github.com/LMFDB/lmfdb/blob/master/GettingStarted.md have been updated to reflect the new configuration.
For more information on the details of how database interface has changed, see the FAQ at https://github.com/LMFDB/lmfdb/blob/master/Postgres_FAQ.md. We have done everything we can to make the transition as seamless as possible, and most developers should notice very little in the way of changes (all the code has already been modified to use the new postgres interface layer described in the FAQ).
For developers who have accounts on atkin or lehner at Warwick that they use to upload or modify large datasets to the mongo-database at Warwick, there is a development machine called legendre at MIT that is now available to play the same role running a copy of the postgres database. In order to get an account on this machine, you will need to send me a public ssh key and I can then grant you access to this machine.
John Jones
Aug 10, 2018, 12:20:27 PM8/10/18
to lmdb
Hi,
Great job in setting up/documenting the transition. Here are my ssh keys (office, home, laptop):
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzbsXHJlegIrZYUlxMUwckdoB9rVQmLkIUTfeOf7hYytDwn1dWzVNPeq3DEBexYheIGLlnbYWBy118XNSfhQnWBt7XBm9APln9VVdGvcU6t3OuFu1VkSKw4VFqq7vPs4j3sBe8bC3wwJTGvxz+s3tItRMJ0y74FqrLpU8ckoIn83XhM6CXY5YkdiIa72PA949HGlYkZHQB1C3FP/3M8COopIa3xiqv2EKdqlXtEi2aXvownQY9A6pEdPUQqqO7a5X1Kyz3x5tmNnYNvhqaPnLEhdwfxawgn7MuCtiKTMRmc+iVmR9bhxQpnDcvYjZnNLGVyDrpnmpBgdSPwkzpyFG9 jj@hobbes
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHsqP5JgfMqkz7CfAY4ZAi4Mh5wdV3E2tAbe/ECsv0TUiGj+zf8e8qqlwWAn8iyUteNFip8FC0kE0EWLW/tKtMa5DQehr0wl8x+Yg8dYQL3JXk3rPHNc+bnLoc2/lx3pk+uNu3Oqlb3Nyrjy5ubcMdlTIn2dgu7vcLv66F3QYr/ed72JoL2XMR7ACzYCbX+FuJ3TsoKZHAXpKLMQGM3S0IsKVKCnWHF//biufsLDkTHFjk8Z7AuFLB45+yQJ02L2NOOptHCCn4z9SN3kmeP+B7kGSZ06v2iqF2lSVppj1KRp65wGEzO2WDAsibYVwXdCBU0Y1GEWCGGEkz0x6ohnMD j...@calvin.local
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCv4xGdjDJFuhUQy+a8hJtPGqm0VSlVTrmEX4VKAT3vDE7tOHUXC0SKJWyJmE7DwXBk4d/8f+Z1+SKaiMNSrpPdx0UnbqG9tAOaTfyPVYHvDghjwFFEfaYSoJwGLXoYYTCBnndwOGpSu5qUo0y+GgTwPmHrC8IdMGlz7FqwRx1acDFCp4YVj3yfafZy92/TghkYDBMCtb+sjWWiD6ooc7unGsDLVSojH2wZAzOpr0V04Oumc9st9xqXkqdS6DGtfvu1CM37ax/J8rsrPXf6uvpBn8CIbWpOrx6Rf9ywtoUzr8RJcvT5Hg2y7KOQkWs0WkDl+grdb7RU8XkDG0bIwby5 jj@serre
John
--
You received this message because you are subscribed to the Google Groups "lmdb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lmdb+unsubscribe@googlegroups.com.
To post to this group, send email to lm...@googlegroups.com.
Visit this group at https://groups.google.com/group/lmdb.
For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages