GSoC Proposal to extend the parallel test runner to Windows and macOS and to support Oracle parallel test running
184 views
Skip to first unread message
Ahmad A. Hussein
Mar 23, 2020, 3:22:43 PM3/23/20
to Django developers (Contributions to Django itself)
Django's parallel test runner works through forking processes, making it incompatible on Windows by default and incompatible in macOS due to a recent update. Windows and macOS both support spawn and have it enabled by default. Databases are cloned for each worker.
To switch from fork to spawn, state setup will be handled by spawned processes instead of through inheritance via fork. Worker’s connections to databases can still be done through get_test_db_clone_settings which changes the names of databases assigned to a specific worker_id; however, SQLite's cloning method is incompatible with spawn.
SQLite’s cloning method relies on it both being in-memory and fork as when we fork the main process, we retain SQLite's in-memory database in the child process. The solution is to get a SQL dump of the database and throw it into the target cloned databases. This is also the established standard in handling MySQL’s cloning process. Both Postgresql's and MySQL's cloning methods are independent of fork or spawn and won't require any modification.
Oracle has not been implemented in the parallel test runner originally even on Linux and I propose to extend support to Oracle as well in my proposal. I want to confirm if there is significant support behind this as a feature or not before I commit to writing a specification, but as a summary it is definitely possible as the work collaborated on by Aymeric Augustin and Shai Berger show that cloning CAN be done through multiple ideas. The reason why it's a headache is that Oracle does not support separate databases under a single user- unlike our other supported databases, so we can't clone databases without introducing another user. Some methods may also need to be rewritten to accommodate for the Oracle backend, but that isn't an issue. I've glossed over writing out a schedule or a more detailed abstract as I I'm mainly posting here to see if there is indeed support for the Oracle proposal and to make sure I am not missing any details in regards to adapting the current parallel test runner to work through spawn. Let me know what you think.
Regards,
Ahmad
Aymeric Augustin
Mar 26, 2020, 4:39:28 AM3/26/20
to django-d...@googlegroups.com
Hello Ahmad,
I believe there's interest for supporting parallel test runs on Oracle, if only to make Django's own test suite faster.
I'm not very familiar with spawn vs. fork. As far as I understand, spawn starts a new process, so you'll have to redo some initialization in the child processes. In a regular application, this would mean calling `django.setup()`. In Django's own test suite, it might be different; I don't know. Try it and see what breaks, maybe?
Hope this helps :-)
--
Aymeric.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/317f67c6-4b23-483f-ada5-9bdbb45d0997%40googlegroups.com.
Tom Forbes
Mar 26, 2020, 7:35:34 AM3/26/20
to django-d...@googlegroups.com
> incompatible in macOS due to a recent update
Can you elaborate on this a bit? How has a recent update made forking incompatible on MacOS?
Tom
Carlton Gibson
Mar 26, 2020, 7:40:03 AM3/26/20
to Django developers (Contributions to Django itself)
Hi Tom.
(I was wondering if you might fancy mentoring on this project? 🙂)
On Thursday, 26 March 2020 12:35:34 UTC+1, Tom Forbes wrote:
> incompatible in macOS due to a recent updateCan you elaborate on this a bit? How has a recent update made forking incompatible on MacOS?
macOS has deprecated the fork launch method, to be removed in a future version.
Python 3.8+ then swaps the default launch method to spawn, so on Python 3.8+ we no longer have parallel test running on the mac.
(For now, fork still actually works, so you can `set_start_method()` to work around this, but...)
Kind Regards,
Carlton
Carlton
Abhijeet Viswa
Mar 26, 2020, 7:45:14 AM3/26/20
to django-d...@googlegroups.com
Hey,
Since Python 3.8, the default method for the multiprocessing module is spawn for Mac OS.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/322804F5-F44F-4EF8-AF33-C460EC509915%40tomforb.es.
--
Abhijeet
Tom Forbes
Mar 26, 2020, 6:02:24 PM3/26/20
to django-d...@googlegroups.com
Ahh, interesting! I think I can help mentoring, I’ll leave a reply in the official thread 💪
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/e5a1464f-553c-4990-8dd2-6c4086e329c4%40googlegroups.com.
Ahmad A. Hussein
Mar 28, 2020, 2:59:18 AM3/28/20
to Django developers (Contributions to Django itself)
Apologies for the late response. I've had to attend to personal matters concerning the current crisis we're all facing. All is well though
I should have posted a more detailed post from the get-go. I apologize for the lack of clarity as well.
Last week, I initially did exactly what you suggested. I called django.setup() in each child process during worker initialization. This fixed app registry issues but like you said it isn't enough for testing. Test apps and test models were missing and caused tons of errors. Later, I read through runtests.py and saw the setup method there; it was exactly what I needed since it searched for setup the correct template backend, searched for test apps and added them to installed apps, and called django.setup(). I passed that method along with its initial arguments into the runner so I could call it during worker initialization. That fixed all errors related to state initialization. I do wonder though if any meaningful time could be saved if we use a cache for setup instead of having to call the setup for each process.
The last glaring hole was correct database connections. I had a naming mismatch with Postgres and I ended up fixing that through prepending "test_" in each cloned database's name during worker initialization. In case of the start method being fork, we can safely ignore that step and it'll work fine.
Currently what I'm figuring out is getting a SQL dump to change SQLite's cloning method and implementing an Oracle cloning method. I'm searching through Django's documentation and internal code to see if there is a ready-made SQL dump method for SQLite and if not I'll search for it in a third-party library, and if I still don't find it, I'll start thinking about a ground-up implementation using the ORM.
As for progress on the Oracle cloning method, I'm consulting Oracle documentation right now to see if anything has changed in the last 5 years. If I don't find anything interesting, I'll start toying around with Shai Berger's ideas to see what works and what's performance-costly.
Lastly, testing does seem to be an enigma to think about right now. I've thought about tests for both SQLite's and Oracle's cloning method, but I can't imagine anything else.
If you have any pointers, suggestions or feedback, I'd love to hear it! And thank you for your help so far.
Regards,
I should have posted a more detailed post from the get-go. I apologize for the lack of clarity as well.
Last week, I initially did exactly what you suggested. I called django.setup() in each child process during worker initialization. This fixed app registry issues but like you said it isn't enough for testing. Test apps and test models were missing and caused tons of errors. Later, I read through runtests.py and saw the setup method there; it was exactly what I needed since it searched for setup the correct template backend, searched for test apps and added them to installed apps, and called django.setup(). I passed that method along with its initial arguments into the runner so I could call it during worker initialization. That fixed all errors related to state initialization. I do wonder though if any meaningful time could be saved if we use a cache for setup instead of having to call the setup for each process.
The last glaring hole was correct database connections. I had a naming mismatch with Postgres and I ended up fixing that through prepending "test_" in each cloned database's name during worker initialization. In case of the start method being fork, we can safely ignore that step and it'll work fine.
Currently what I'm figuring out is getting a SQL dump to change SQLite's cloning method and implementing an Oracle cloning method. I'm searching through Django's documentation and internal code to see if there is a ready-made SQL dump method for SQLite and if not I'll search for it in a third-party library, and if I still don't find it, I'll start thinking about a ground-up implementation using the ORM.
As for progress on the Oracle cloning method, I'm consulting Oracle documentation right now to see if anything has changed in the last 5 years. If I don't find anything interesting, I'll start toying around with Shai Berger's ideas to see what works and what's performance-costly.
Lastly, testing does seem to be an enigma to think about right now. I've thought about tests for both SQLite's and Oracle's cloning method, but I can't imagine anything else.
If you have any pointers, suggestions or feedback, I'd love to hear it! And thank you for your help so far.
Ahmad
--Aymeric.
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.
Adam Johnson
Mar 29, 2020, 11:56:20 AM3/29/20
to django-d...@googlegroups.com
Currently what I'm figuring out is getting a SQL dump to change SQLite's cloning method and implementing an Oracle cloning method. I'm searching through Django's documentation and internal code to see if there is a ready-made SQL dump method for SQLite and if not I'll search for it in a third-party library, and if I still don't find it, I'll start thinking about a ground-up implementation using the ORM.
SQLite has an online backup API and the "VACUUM INTO" command both of which can efficiently clone databases: https://sqlite.org/backup.html . I think they can even copy memory-to-memory with the right arguments.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1e185df6-e6ea-4066-84b5-c2e0a150239a%40googlegroups.com.
--
Adam
Ahmad A. Hussein
Mar 30, 2020, 3:13:43 AM3/30/20
to django-d...@googlegroups.com
Thank you Adam for the amazing pointer. That's exactly the solution I was looking for.
I didn't even know VACUUM INTO was a thing in SQLite3 (quite nifty). Both sources you listed can definitely copy memory-to-memory efficiently. I had also found two useful APIs in the standard library's documentation: iterdump() and backup(). The latter is an attempt at recreating the online backup API you linked.
I did face an issue in my implementation though. Memory-to-memory copying is impossible under the current conditions.
We can't copy memory-to-memory and retain the resultant databases into our spawned workers since spawned child processes don't share an environment with parent processes.
There are two possible workarounds:
I didn't even know VACUUM INTO was a thing in SQLite3 (quite nifty). Both sources you listed can definitely copy memory-to-memory efficiently. I had also found two useful APIs in the standard library's documentation: iterdump() and backup(). The latter is an attempt at recreating the online backup API you linked.
I did face an issue in my implementation though. Memory-to-memory copying is impossible under the current conditions.
We can't copy memory-to-memory and retain the resultant databases into our spawned workers since spawned child processes don't share an environment with parent processes.
There are two possible workarounds:
1. To copy our in-memory database into an on disk database using VACUUM INTO and subsequently restore it into in-memory during worker initialization
2. Pass a string query resulting from calling Connection.iterdump() into worker initialization and call that to restore copies of the database into in-memory for every child process.
I'm testing out both now to see if there are any major differences. There might be a performance difference between them. I'm also checking if there are any unexpected errors that might arise.
Regards,
I'm testing out both now to see if there are any major differences. There might be a performance difference between them. I'm also checking if there are any unexpected errors that might arise.
Regards,
Ahmad
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMyDDM1YU51rQZHARD0NRNKDyY0mEnt8XDPWAyyp%3DbPeT0bsqg%40mail.gmail.com.
Tom Forbes
Mar 30, 2020, 3:39:13 AM3/30/20
to django-d...@googlegroups.com
There is an interesting addition to the standard library in Python 3.8: multiprocessing.shared_memory (https://docs.python.org/3/library/multiprocessing.shared_memory.html). It’s only on 3.8 but it might be worth seeing if that has a performance improvement over passing a large string into the workers init process.
Have you also considered simply re-running the migrations and creating the SQLite database from scratch in each worker, or simply copying the SQLite database file itself?
On 30 Mar 2020, at 08:13, Ahmad A. Hussein <ahmadah...@gmail.com> wrote:
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAJNa-uNa8rCj_d%2BLt00HceLP97QmbPM6UTRV%3DqpAmmgu8pC23w%40mail.gmail.com.
Ahmad A. Hussein
Mar 30, 2020, 5:35:09 AM3/30/20
to django-d...@googlegroups.com
I did stumble upon shared memory when I originally wanted to pass the database connection itself into shared memory. Your idea of passing the query string is MUCH more reasonable, can be done using shared memory, and will ultimately be much more cleanly written.
i'll definitely test out if it has significant performance boost and see how it compares to other methods.
I did consider copying the SQLite database file itself. That's essentially what's happening with VACUUM INTO. It's a SQLite command that lets you clone a source database whether in-memory or on disk into a database file. I got the parallel test runner to work with these database files, creating a database file for each worker. I think there's a performance boost to have if we make the databases for workers in-memory, especially if the test suite and tables grow in number; however, we also need to consider the time it takes to make the databases in-memory.
Re-running migrations and recreating the SQLite database from scratch for every worker might be a time-sink, but it could actually prove quicker than other methods. I'll try it out as well and see.
Thank you for your suggestions
Regards,
Ahmad
i'll definitely test out if it has significant performance boost and see how it compares to other methods.
I did consider copying the SQLite database file itself. That's essentially what's happening with VACUUM INTO. It's a SQLite command that lets you clone a source database whether in-memory or on disk into a database file. I got the parallel test runner to work with these database files, creating a database file for each worker. I think there's a performance boost to have if we make the databases for workers in-memory, especially if the test suite and tables grow in number; however, we also need to consider the time it takes to make the databases in-memory.
Re-running migrations and recreating the SQLite database from scratch for every worker might be a time-sink, but it could actually prove quicker than other methods. I'll try it out as well and see.
Thank you for your suggestions
Regards,
Ahmad
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/5217A362-5132-4319-9F1F-E8EBA049CB4E%40tomforb.es.
Tom Forbes
Mar 30, 2020, 5:46:39 AM3/30/20
to django-d...@googlegroups.com
Awesome! However there is always a tradeoff between speed and code complexity - it might not be worth a more complex system involving shared memory or copying SQLite files in other ways if it only saves ~5 seconds at the start of a test run compared to re-running the migrations in each worker. I guess this might be unique to SQLite where things are generally faster.
Just something to keep in mind, I’m excited to see the results though!
Tom
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAJNa-uPOr_v8ggOCyNh09NARSWXFf-gGvbnHoG3nSxft_SbHZg%40mail.gmail.com.
Ahmad A. Hussein
Mar 30, 2020, 8:13:45 AM3/30/20
to django-d...@googlegroups.com
You're definitely right!
I just finished rewriting it to re-run migrations and setup in each worker. I didn't bench mark it so I'll do that again later, but as a gut reaction it isn't significantly slower; but of course the numbers will tell the real story.
Passing the dump string through shared memory or converting in-memory to on disk seems way too overkill now. The performance drop probably won't be significant especially since this is SQLite.
I'll update you here with the exact benchmarks later. I'm busy now with school work, but I'll get back to it again later today.
It is definitely exciting to find out!
Regards,
I just finished rewriting it to re-run migrations and setup in each worker. I didn't bench mark it so I'll do that again later, but as a gut reaction it isn't significantly slower; but of course the numbers will tell the real story.
Passing the dump string through shared memory or converting in-memory to on disk seems way too overkill now. The performance drop probably won't be significant especially since this is SQLite.
I'll update you here with the exact benchmarks later. I'm busy now with school work, but I'll get back to it again later today.
It is definitely exciting to find out!
Regards,
Ahmad
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/8743CDFB-32AF-477F-8AE5-514A8E953159%40tomforb.es.
Adam Johnson
Mar 30, 2020, 2:25:05 PM3/30/20
to django-d...@googlegroups.com
I just finished rewriting it to re-run migrations and setup in each worker. I didn't bench mark it so I'll do that again later, but as a gut reaction it isn't significantly slower; but of course the numbers will tell the real story.
I'm a little concerned about rerunning migrations in each worker in case they are not deterministic (for some maybe slightly odd projects).
Also I'd expect in many projects a dump and restore will be faster than running migrations.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAJNa-uOi-6NYi7i_QN9axQidwWd%3Dqq924afcWHHihvHnXd%3DpHw%40mail.gmail.com.
--
Adam
charettes
Mar 30, 2020, 2:41:43 PM3/30/20
to Django developers (Contributions to Django itself)
I agree with Adam here, it not just about running the SQL itself; the current migration framework require models to be (re-)rerendered before a lot of schema operations which slows down dramatically as project grows.
The dump/restore approach will undeniably be faster than running migrations for each test process.
Simon
Tom
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1e185df6-e6ea-4066-84b5-c2e0a150239a%40googlegroups.com.
--Adam--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMyDDM1YU51rQZHARD0NRNKDyY0mEnt8XDPWAyyp%3DbPeT0bsqg%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAJNa-uNa8rCj_d%2BLt00HceLP97QmbPM6UTRV%3DqpAmmgu8pC23w%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/5217A362-5132-4319-9F1F-E8EBA049CB4E%40tomforb.es.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAJNa-uPOr_v8ggOCyNh09NARSWXFf-gGvbnHoG3nSxft_SbHZg%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/8743CDFB-32AF-477F-8AE5-514A8E953159%40tomforb.es.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAJNa-uOi-6NYi7i_QN9axQidwWd%3Dqq924afcWHHihvHnXd%3DpHw%40mail.gmail.com.
--Adam
Tom Forbes
Mar 30, 2020, 4:01:40 PM3/30/20
to django-d...@googlegroups.com
You’re both most likely correct, but I think it’s good to get some numbers and compare implementation complexity.
It’s also worth considering that often as a project grows in complexity the time it takes to do a full test run grows with it, so perhaps the overhead would be in a fairly constant (and reasonable) percentage of overall runtime.
Or maybe not! I just think we should benchmark the naive implementation along with the others.
Tom
On 30 Mar 2020, at 19:41, charettes <chare...@gmail.com> wrote:
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/017dbc0f-cc05-4bbc-9ec1-2676a64693fd%40googlegroups.com.
Abhijeet Viswa
Mar 30, 2020, 4:12:15 PM3/30/20
to django-d...@googlegroups.com
Hi,
I'd like to point out that the MySQL backend uses the mysqld utility to create a database dump which is then imported. So, as far as implementation complexity goes, I believe migration once again during setup of the child process might complicate it further. It might just be easier to use on-disk database for the child processes. That being said, benchmarking is never a bad thing.
Also, I would highly appreciate it if y'all could check out my proposal on the same topic. I had put it up on the Django Forums last week and on the mailing list a couple of days ago. Only Carlton had replied. I know there's less than 24 hours left to submit the proposal, but any feedback will help.
Link to the proposal: https://gist.github.com/abhijeetviswa/08c23388ae3d317621e23b947594b9f3
Link to the forum post: https://forum.djangoproject.com/t/gsoc-2020-my-proposal-for-a-parallel-test-runner/1518
Thanks.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/86226E01-7850-4CE8-BF11-E229FEE6AEC9%40tomforb.es.
--
Abhijeet
Reply all
Reply to author
Forward
0 new messages