turn on "string_if_invalid" as a ‘development default’

[guettli]

I would like to do this:

  turn on "string_if_invalid" as a ‘development default’

The django docs say:

 While string_if_invalid can be a useful debugging tool, it is a bad idea to turn it on as a ‘development default’.

https://docs.djangoproject.com/en/1.9/ref/templates/api/#how-invalid-variables-are-handled

We were hit by some nasty bugs in production environment which were not detected in our continous integration plattform.

If "string_if_invalid" (or former TEMPLATE_STRING_IF_INVALID) would have been activated,
we would have noticed the bug, and the broken code would have gone live.

How do you handle this?

[James Schneider]

Why not have a second batch of tests that use a settings.py file with a non default value for "string_if_invalid" that runs if the default settings file passes? I've never dealt with a CI platform, but I'm assuming you can do something like that.

- James

[guettli]

A second batch of tests?

I am unsure if I understood you: Do you mean new tests, or running the same
tests again, with a different settings file?

If the first: write new tests: How should these tests be different from the existing tests?

If the second: Why not run the test with the settings which have string_if_invalid enabled in the first run?
What do you gain by running the tests twice?

Regards,
  Thomas Güttler

[James Schneider]

A second batch of tests?

I am unsure if I understood you: Do you mean new tests, or running the same
tests again, with a different settings file?

I meant the latter. Sorry for not being clear.

 

If the first: write new tests: How should these tests be different from the existing tests?

If the second: Why not run the test with the settings which have string_if_invalid enabled in the first run?
What do you gain by running the tests twice?

The two sets of tests could potentially provide two sets of differing results. The development setting you want to change is known to negatively interact with templates used by the Django admin and probably third-party applications that provide templates, per the link that you provided. With that setting changed, you are testing an application that intentionally behaves differently than what will be seen in production. That makes me uncomfortable.

For me, that means running two sets of tests. One of them will closely match the production settings (probably the settings file you're using now), and the other can be tweaked to potentially catch edge-cases or other uncommon bugs that you can/have run in to, or other failure points that may not be easily exposed by the standard settings file. Perhaps the second customized set of tests may not be something that you build into your CI, but may be part of a manual examination performed at various milestones. It would depend on your workflow and CI process, and how often you think these particular bugs may present themselves. Maybe only as a pre-release final check?

Or, if you don't use the admin or any 3rd party applications, you may be safe in enabling it. I personally wouldn't unless I was also doing so in production.

-James

[guettli]

Am Montag, 18. Januar 2016 12:02:15 UTC+1 schrieb James Schneider:

A second batch of tests?

I am unsure if I understood you: Do you mean new tests, or running the same
tests again, with a different settings file?

I meant the latter. Sorry for not being clear.

 

If the first: write new tests: How should these tests be different from the existing tests?

If the second: Why not run the test with the settings which have string_if_invalid enabled in the first run?
What do you gain by running the tests twice?

The two sets of tests could potentially provide two sets of differing results. The development setting you want to change is known to negatively interact with templates used by the Django admin and probably third-party applications that provide templates, per the link that you provided. With that setting changed, you are testing an application that intentionally behaves differently than what will be seen in production. That makes me uncomfortable.

Yes, I understand "...behaves differently than what will be seen in production. That makes me uncomfortable." I see it the same.

I would like  string_if_invalid to raise an exception immediately. Maybe even in prod, since: "...behaves differently than what will be seen in production. That makes me uncomfortable."

Zen of Python:

> Errors should never pass silently.
 

For me, that means running two sets of tests. One of them will closely match the production settings (probably the settings file you're using now), and the other can be tweaked to potentially catch edge-cases or other uncommon bugs that you can/have run in to, or other failure points that may not be easily exposed by the standard settings file. Perhaps the second customized set of tests may not be something that you build into your CI, but may be part of a manual examination performed at various milestones. It would depend on your workflow and CI process, and how often you think these particular bugs may present themselves. Maybe only as a pre-release final check?

We have no "pre-release final check/test". We run the whole test-suite twice daily. This is done automatically be jenkins.
We try to automate as much as possible. Manual examination is possible, but who wants to do this? I would like to have it automated.
All tests run in around 10 minutes.

I guess we will see bugs like  four times per year.
With "like this" I mean:

 * bugs caused by the template engine returning the empy string instead of an exception
 * bugs which are not detected by CI and get into production

 

Or, if you don't use the admin or any 3rd party applications, you may be safe in enabling it. I personally wouldn't unless I was also doing so in production.

Yes, dev and prod should not be much different

 Thank you for your input.

Regards,
  Thomas

[James Schneider]

We have no "pre-release final check/test". We run the whole test-suite twice daily. This is done automatically be jenkins.
We try to automate as much as possible. Manual examination is possible, but who wants to do this? I would like to have it automated.
All tests run in around 10 minutes.

Hence the reason I mentioned that it will be dependent on your workflow. If this is something you want to automate, obviously you'll need to figure out how to get it to work with your CI implementation.

 

I guess we will see bugs like  four times per year.

With "like this" I mean:

 * bugs caused by the template engine returning the empy string instead of an exception
 * bugs which are not detected by CI and get into production

I did a bit of poking around and I found this:

https://docs.djangoproject.com/en/dev/topics/testing/tools/#overriding-settings 

It actually looks like it wouldn't be too difficult to take specific existing tests and re-run them in the same test run with the string_if_invalid tweaked. That would save you from having to modify your CI, and would provide coverage in both a 'production mirror' state and the alternative 'template bug catcher' state. You can probably subclass your existing test classes and just decorate them appropriately, although you might have other things you're looking for in those cases. 

Never tried it myself, but looks to be worth a shot in your case.

-James