LiveServerTestCase change in Django 1.11 means can't run functional tests against external server?

[Andrew Wall]

Very much appreciate the Django framework. 

I noticed in the docs for Django 1.11 that the DJANGO_LIVE_TEST_SERVER_ADDRESS environmental variable is slated to be removed, along with the accompanying
python manage.py test --liveserver option.  I'm concerned that without the ability to specify a remote IP address using --liveserver, it will no longer be possible to run functional tests using selenium against an external server.  I learned this technique from Harry Percival's excellent TDD with Python (see http://chimera.labs.oreilly.com/books/1234000000754/ch08.html#_configuring_domains_for_staging_and_live), where the test command is called from the local machine but you pass in the external IP.  Will there be a different way to do this in Django 1.11? Perhaps the change to bind LiveserverTestCase to port zero by default can be made while retaining the option to pass in --liveserver?  Realize the release is a ways away but would appreciate any help as I've come to rely on this method to test server deployments.  Thank you!

[Tim Graham]

At the time of the implementation [0], no one raised that use case. Probably we can reopen the ticket and add it back.

[0] https://groups.google.com/d/topic/django-developers/_TD8IkSLgqE/discussion

[Alex Riina]

Thanks for including a link to the TDD tutorial!

As far as I can tell, the author wants to use `--liveserver` to point his functional tests at an external server so he can run his tests through nginx after running the same tests in his normal test environment. Based on the shell prompts in http://chimera.labs.oreilly.com/books/1234000000754/ch08.html#_creating_the_database_with_migrate I think he intends to have the database and code running on his staging server while he runs his tests locally. He also avoids the TransactionTestCase truncation operations by shorting out the LiveServerTestCase setUp / tearDownClass.

Simplest way?

The simplest way I can think of to resolve the issue is to read the external liveserver out of os.env instead of sys.argv. Then you can follow along exactly.

$ export EXTERNAL_SERVER='http://superlists-staging.ottg.eu'
$ python manage.py test

`allow_database_queries = False` would make me trust the setUpClass / tearDownClass overrides more but my main issue with this approach is that it does a lot of extra work: 

1. run tests locally
   
2. deploy to your code to your staging server
   
3. runserver with fixed port on staging server
   
4. run tests locally with LiveServerTestCases running against staging server

It would be nice for the last run-through to _only_ run the functional tests and not have to setUp the local database or run the other tests against your local branch.

Make unittest run functional tests remotely

# app/test_functional_something.py
import os

if 'DJANGO_SETTINGS_MODULE' in os.env:
    from django.tests import StaticLiveServerTestCase
    FunctionalTest = StaticLiveServerTestCase
else:
    class FunctionalTest(unittest.TestCase):
        live_server_url = 'http://superlists-staging.ottg.eu'

then you can either run the tests via the DiscoveryRunner on the local database

$ python manage.py test

or via the normal unittest runner on your remote server

$ python -m unittest discover -p 'test_functional_*.py'

Unfortunately, you'll lose a lot of power in not being able to reuse your form code, assert anything about the database state on the remote server, access django assertion methods, ... 

At that point, I'd probably just run the tests on the remove server and do something like

Hardcode the port

class FixedPortServerThread(LiveServerThread):
    def _create_server(self, port):
         if os.env.get('THROUGH_NGINX'):
             return super(FixedPortServerThread, self)._create_server(port=8000)
         else:
             return super(FixedPortServerThread, self)._create_server(port=port)


class MyLiveServerTestCase(LiveServerTestCase):
    server_thread_class = FixedPortServerThread

On Saturday, October 15, 2016 at 9:32:54 AM UTC-4, Andrew Wall wrote:

[Tim Graham]

Good points -- given the existing subclassing needed to get LiverServerTestCase to support that use case, it might be reasonable not to worry about backwards-compatibility here. I'd be interested to hear Harry's (the author) thoughts about it.

[Harry Percival]

Hi everyone, thanks for cc'ing me!

I'm relaxed about this -- my use of the --liveserver command-line arg was always a hack, and i'm happy to figure out a different hack as and when i come to update the book for 1.11.   A custom test runner that only does FTs, and maybe hacks the class to no longer do the django stuff would probably do it...

i wouldn't do anything special on my account.  do keep me posted if anyone does decide to write their own hack and feels like sharing it tho!

hp

(PS not sure if i'm on the django-dev mailing list, do forward my email if necessary)

--

--
Harry Percival
+44 78877 02511

[Andrew Wall]

Thanks everyone.  I guess my Django skills aren't advanced enough to figure out how this will work without the --liveserver command.

Would it be easy to retain this command-line arg or would it require a significant change to the implementation?  It seems like a useful command.

[Tim Graham]

Take a look at the commit that made the removal:
https://github.com/django/django/commit/81cdcb66bc74a0768d13f0e18872d46739028e64

--liverserver was only a shortcut for setting the DJANGO_LIVE_TEST_SERVER_ADDRESS environment variable, so that could be done without a command line argument, however, the environment variable also isn't consulted anymore. This could be added back in some fashion, though it's unclear to me if the use case of running LiveServerTestCase against an external server is a use case that justifies this.

[Alex Riina]

I use --liveserver for a slightly different case and had missed the original conversation. I'm not sure if mine justifies the code either, but for consideration:

My company's authentication layer includes a homebrewed google oauth2 endpoint. I've added localhost:8000 to the valid redirect_uris and use selenium to handle a few different login cases. I'd be surprised if I'm the only one using a setup like this to avoid copying around inscrutable json blobs.

Unfortunately the tests are already hard to keep running (frequently entering a security codes via the selenium driver on CI servers) so I've been planning to ditch them.

[Francis Mwangi]

please reopen the ticket, we are having a pain testing code that generating code. For instance

class A(object):

      url = "some_url"

test.py

class TestingA(LiveserverTestcase):

      def test_a()
            "we can't override the url defined in object A since its initialized even before we a have access to self.liveserver_url"

[Tom Forbes]

Shouldn't A.url be a relative URL in this case?

I agree the ticket should be re-opened though, we had some pains working around this, but it was surmountable.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/b74d3aa2-a033-4f98-a558-cdbbd68717d3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Graham]

Maybe the LiveServerTestCase.port attribute added in Django 1.11.2 will help, https://code.djangoproject.com/ticket/28212.

At this point, the original change is released, so we would open a new ticket for further changes.

On Thursday, June 1, 2017 at 10:50:54 AM UTC-4, Tom Forbes wrote:

Shouldn't A.url be a relative URL in this case?

I agree the ticket should be re-opened though, we had some pains working around this, but it was surmountable.

On 1 Jun 2017 15:31, "Francis Mwangi" <mw...@tumacredo.com> wrote:

please reopen the ticket, we are having a pain testing code that generating code. For instance

class A(object):

      url = "some_url"

test.py

class TestingA(LiveserverTestcase):

      def test_a()
            "we can't override the url defined in object A since its initialized even before we a have access to self.liveserver_url"

On Saturday, October 15, 2016 at 5:13:07 PM UTC+3, Tim Graham wrote:

At the time of the implementation [0], no one raised that use case. Probably we can reopen the ticket and add it back.

[0] https://groups.google.com/d/topic/django-developers/_TD8IkSLgqE/discussion

On Saturday, October 15, 2016 at 9:32:54 AM UTC-4, Andrew Wall wrote:

Very much appreciate the Django framework. 

I noticed in the docs for Django 1.11 that the DJANGO_LIVE_TEST_SERVER_ADDRESS environmental variable is slated to be removed, along with the accompanying
python manage.py test --liveserver option.  I'm concerned that without the ability to specify a remote IP address using --liveserver, it will no longer be possible to run functional tests using selenium against an external server.  I learned this technique from Harry Percival's excellent TDD with Python (see http://chimera.labs.oreilly.com/books/1234000000754/ch08.html#_configuring_domains_for_staging_and_live), where the test command is called from the local machine but you pass in the external IP.  Will there be a different way to do this in Django 1.11? Perhaps the change to bind LiveserverTestCase to port zero by default can be made while retaining the option to pass in --liveserver?  Realize the release is a ways away but would appreciate any help as I've come to rely on this method to test server deployments.  Thank you!

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.