When tests pass in CI but fail in the development environment

[Jason Hsu]

I understand that if your app's tests fail in the continuous integration environment but pass in your development environment, you should figure out why tests fail in CI and NOT blow off the matter.

What should you do when the reverse is true - when tests pass in CI but not in your local development environment?  Do you blow off testing in the local environment, push your changes to an alternate branch, and see if things pan out in the CI environment?  Maybe this is appropriate in some situations, but it sounds like a cop-out to me.

[Brandon McClelland]

The same logic for local-PASS and CI-FAIL applies here for local-FAIL and CI-PASS: the main purpose of testing in various environments is to avoid "it works on my machine!" 

Try reading some of the results from googling 

test 'fail locally' but pass on CI -"pass locally" 

and you should find that when people investigated the cause of the discrepancy between environments they usually learned how to improve their code and remove bugs by not ignoring this. Good luck!

On Wed, Jan 23, 2019 at 3:12 PM Jason Hsu <jhsu8...@gmail.com> wrote:

I understand that if your app's tests fail in the continuous integration environment but pass in your development environment, you should figure out why tests fail in CI and NOT blow off the matter.

What should you do when the reverse is true - when tests pass in CI but not in your local development environment?  Do you blow off testing in the local environment, push your changes to an alternate branch, and see if things pan out in the CI environment?  Maybe this is appropriate in some situations, but it sounds like a cop-out to me.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-ta...@googlegroups.com.
To post to this group, send email to rubyonra...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/3b5c0b69-639b-4c93-b79f-6d3e461bbd4c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brandon McClelland

User Support Technician

Steve Jackson Games

[Rob Zolkos]

I would check the environments are the same and try to diagnose what is different.

Start with 

- Same Ruby versions

- Same database and database versions (especially if the failures are database ones)

Is the failures due to an area of code that uses any native gems that may have not been compiled correctly.

On Thu, Jan 24, 2019 at 7:12 AM Jason Hsu <jhsu8...@gmail.com> wrote:

I understand that if your app's tests fail in the continuous integration environment but pass in your development environment, you should figure out why tests fail in CI and NOT blow off the matter.

What should you do when the reverse is true - when tests pass in CI but not in your local development environment?  Do you blow off testing in the local environment, push your changes to an alternate branch, and see if things pan out in the CI environment?  Maybe this is appropriate in some situations, but it sounds like a cop-out to me.

--