How to get the number of tests excluded by `c.filter_run_excluding`?

[Jan P.]

RSpec has this nice method to exclude individual tests/examples or whole groups by using filter_run_excluding in the config, then tagging the examples:

https://relishapp.com/rspec/rspec-core/v/3-7/docs/filtering/exclusion-filters

RSpec.configure do |c|
  c.filter_run_excluding :skip => true
end

RSpec.describe "something" do
  it "does one thing" do
  end

  it "does another thing", :skip => true do
  end
end

"does one thing" will be checked,
"does another thing" will not.

We are using this, for example, to skip some tests depending on the platform the test is run on by wrapping the c.filter_run_excluding :skip => true in an if block:

If Mac, 
   no exclusions, 
if Ubuntu, 
   exclude tests that do something with Xcode.

Right now the numbers of passing examples/test is just lower if the exclusion filter is used, but it would be nice to see the actual number of tests that are skipped.

Is there a way to get the number of tests skipped by this method during a test run?

Thanks,

Jan

[Myron Marston]

RSpec does not provide a way to get the number of examples that were excluded by its inclusion or exclusion filters, but there’s a different mechanism that will do what you want. Instead of filtering the examples (which excludes them from consideration entirely), you can skip them, which prevents the body of the example from running, sets the example’s status to :pending, will print the example in yellow in the formatter output, and will count the example in the summary total printed at the end (e.g. “500 examples, 0 failures, 20 pending”). Normally, :skip metadata will cause an example to be skipped, but you’ve overwritten it to cause :skip to cause examples to be filtered out.

Here’s my suggestion for how to wire this up.

First, tag any examples that depend upon xcode with :uses_xcode (rather than :skip), e.g.:

it "uses a feature of xcode", :xcode do
  # ...
end

it "does not use xcode at all" do
  # ...
end

Then use define_derived_metadata to automatically tag these examples with :skip if you are not running on OS X:

# spec_helper.rb
require 'rbconfig'

RSpec.configure do |config|
  unless RbConfig::CONFIG['host_os'] =~ /darwin/
    config.define_derived_metadata(:xcode) do |meta|
      meta[:skip] = "Can only be run on OS X"
    end
  end
end

The “Can only be run on OS X” bit will be printed in the output as the reason the examples are pending.

HTH,
Myron

​

--
You received this message because you are subscribed to the Google Groups "rspec" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rspec+unsubscribe@googlegroups.com.
To post to this group, send email to rs...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/3108ef8e-303d-425b-9b00-ab83dfec7633%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jan P.]

Thanks for the quick answer.

I missed "skipping examples" because I was so happy to have found exclusion filters. Sounds like pretty much what I am looking for - even better with the explicit reason I can set for skipping. Will try and report back.

Best,

Jan

To unsubscribe from this group and stop receiving emails from it, send an email to rspec+un...@googlegroups.com.

[Jan P.]

Yes, that works mostly like expected:

4704 examples, 21 failures, 154 pending

and 

...
[18:39:20]: ▸ Pending: (Failures listed here are expected and do not affect your suite's status)

[18:39:20]: ▸ 1) Fastlane Fastlane::EnvironmentPrinter contains main information about the stack

[18:39:20]: ▸ # Requires Xcode to be installed which is not possible on this platform

[18:39:20]: ▸ # ./fastlane/spec/env_spec.rb:28

[18:39:20]: ▸ 2) Fastlane Fastlane::EnvironmentPrinter FastlaneCore::Helper.xcode_version cannot be obtained contains stack information other than Xcode Version

[18:39:20]: ▸ # Requires Xcode to be installed which is not possible on this platform

[18:39:20]: ▸ # ./fastlane/spec/env_spec.rb:47

...

Awesome!

The 21 failures are new though.

I have a _spec.rb file with 21 examples that has a before(:all) that seems to have been filtered with my old solution, but with skip is now executed and fails:

describe Scan do

describe Scan::XCPrettyReporterOptionsGenerator do

before(:all) do

.. code that fails when executed on non-macOS ...

end

describe "xcpretty reporter options generation" do

it "generates options for the junit tempfile report required by scan", requires_xcodebuild: true do

...

Any idea what I can do about this?

-J

[Jan P.]

Excuse the terrible code formatting, seems Google Groups wanted to be helpful.

Another try:

describe Scan do
  describe Scan::XCPrettyReporterOptionsGenerator do
    before(:all) do
      // code that fails when executed on non-macOS
    end

    describe "xcpretty reporter options generation" do
      it "generates options for the junit tempfile report required by scan", requires_xcodebuild: true do
        ...

-J

[Jon Rowe]

You should be able to specify `requires_xcodebuild: true` on the describe as we’ll to prevent the before all being triggered

Jon Rowe

---------------------------

ma...@jonrowe.co.uk

jonrowe.co.uk

To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/bcfd5676-92bd-4051-a3dd-5cb942784698%40googlegroups.com.

[Jan P.]

Jon, I tried 

describe "xcpretty reporter options generation", requires_xcodebuild: true do

and

before(:all), requires_xcodebuild: true do

and

describe Scan::XCPrettyReporterOptionsGenerator, requires_xcodebuild: true do 

but all didn't work and the tests were logged as failure because of the `before` being executed.

Am I doing this wrong somehow?

Code (last iteration) is here: 
https://github.com/fastlane/fastlane/blob/janpio-mark_skipped_tests_as_pending_with_reason/scan/spec/xcpretty_reporter_options_generator_spec.rb
https://github.com/fastlane/fastlane/blob/janpio-mark_skipped_tests_as_pending_with_reason/spec_helper.rb#L68-L83
Matching Circle CI run:
https://circleci.com/gh/fastlane/fastlane/12586

-Jan

[Myron Marston]

Unfortunately, there’s no way to use :skip metadata at the group level to skip before(:all) hooks. That’s because of how metadata in RSpec is modeled, and how RSpec implements :skip. :skip metadata is handled at the level of individual examples, and metadata is inherited from a group to its enclosing examples. So, for example, this works:

RSpec.describe MyClass, :skip do
  it("is skipped"){ }
  it("is also skipped") { }
end

Both examples will be skipped here.

But consider that you can also do this:

RSpec.describe MyClass, :skip do
  it("is skipped"){ }
  it("is also skipped") { }
  it("is not skipped", skip: false) { }
end

Here we have on example that is overriding the skip: true metadata inherited from the group. As a result, a before(:all) hook can’t simply look at the group metadata to decide whether or not to skip or not. In general, before(:all) hooks have lots of gotchas like this because they don’t really fit well into the per-example semantics of so many parts of RSpec. If you can refactor your tests to no longer need such a hook (possibly using a aggregate_failures), that is worth considering. Besides that, the other option you can do is to manually call the skip method from your before(:all) hook:

before(:all) do
  skip "reason to skip" if should_skip?
  # rest of your hook logic
end

HTH,
Myron

​

To unsubscribe from this group and stop receiving emails from it, send an email to rspec+unsubscribe@googlegroups.com.

To post to this group, send email to rs...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/715c3e8b-c8e0-4c98-a0ee-9ebf1d9c4ae4%40googlegroups.com.

[Jan P.]

Thanks Myron, that worked!

> 4704 examples, 0 failures, 175 pending

With code:

    before(:all) do
      skip "Requires `xcodebuild` to be installed which is not possible on this platform" unless FastlaneCore::Helper.is_mac?

As you can see right now I duplicate the reason text, and use the condition that also triggers `skip`/metadata block:
https://github.com/fastlane/fastlane/blob/c6b1ac4621941b2efef702b923a572357c64eea9/spec_helper.rb#L68-L84

Is there somehow a better way to "connect" the skip in `before` to the filters/metadata defined here?
It would be nicer if I could e.g. call a function with the "metadata" ("xcodebuild") as parameter to trigger the skip or something. Any idea?

Best,

Jan

[Myron Marston]

There is a way you can simplify this further:

module HookOverrides
  def before(*args)
    super unless metadata[:skip]
  end
end

RSpec.configure do |c|
  c.extend HookOverrides
end

This overrides before so that it’s a no-op if :skip metadata is set on the example group. With that in place, you don’t need to call skip from your before(:all) hooks. But bear in mind that if you ever set :skip on an example group (to set a default for the group) and then set skip: false for a specific example in the group…this override will cause the before hook to be skipped even though you’d probably expect it in that case. (Which is why we can’t apply this as a generic patch to RSpec itself).

HTH,
Myron

​

To unsubscribe from this group and stop receiving emails from it, send an email to rspec+unsubscribe@googlegroups.com.

To post to this group, send email to rs...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/bb687ad6-5489-4342-b1c8-c1656b9bf2f3%40googlegroups.com.

[Jan P.]

That sounds perfect for my use case. As I introduced `:skip` into the codebase, I would be pretty comfortable with adding this as well (and confine it to non-macOS platforms to limit exposure).

But I can't get it to work. The `before` just runs anyway:

https://circleci.com/gh/fastlane/fastlane/12612

https://github.com/fastlane/fastlane/blob/a103c655c4162723b96cae8f4a8b1258a1419321/spec_helper.rb#L90-L98

(Tried integrating it into my existing RSpec.configure block as well: https://github.com/fastlane/fastlane/pull/11284/commits/1815b4bca04f58ec84fd365097f61b8dd4e68b89#diff-82c4f19cbedb2b12a41b04cd4444aa3cR87)

What am I missing?

Jan

[Myron Marston]

The problem is that you have only set :requires_xcodebuild on the individual examples, and not on the example group as a whole. So when the example group is defined, there is no :requires_xcodebuild metadata, and the define_derived_metadata block does not get invoked to set :skip. Then when you call before, metadata[:skip] returns false, so it proceeds to define the hook normally.

RSpec examples inherit metadata from their parent example group (and enclosing example groups inherit from their parent group), so my suggestion is to remove :requires_xcodebuild from the individual examples in that file, and set it on the example group instead. That’ll both fix the issue and ensure that every example automatically has that metadata.

Myron

​

To unsubscribe from this group and stop receiving emails from it, send an email to rspec+unsubscribe@googlegroups.com.

To post to this group, send email to rs...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/5896f17f-84f3-48da-86f1-b99924e2a5ba%40googlegroups.com.

[Jan P.]

Thanks for the explanation, now it makes sense what was happening.

I applied that to our project and it is working as expected.
Thank you so much for your help.


Can a `before` hook ever access the metadata of an example (`before(:each)`?), or does this only get set after `before` is already finished executing?
I am lucky to not have a test file right now where I have to differentiate between tests with a `before` that should run and tests that shold not - but I am sure this will happen sooner or later :/

Jan

[Myron Marston]

Can a before hook ever access the metadata of an example (before(:each)?)

A before(:each)/before(:example) hook can access the metadata of the example, but a before(:all)/before(:context) hook cannot, because in the latter case, there is no specific example the hook is running as a part of. It’s another reason to generally avoid :all/:context hooks.

To access the metadata from a hook, you just receive an example argument, which has a metadata method:

before(:example) do |example|
  pp example.metadata
end

HTH,
Myron

​

To unsubscribe from this group and stop receiving emails from it, send an email to rspec+unsubscribe@googlegroups.com.

To post to this group, send email to rs...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/f001d009-28cb-4be2-9c41-2f8fa29893a1%40googlegroups.com.