best practices, test::unit, cassette per test? erasing cassettes?
930 views
Skip to first unread message
Jonathan Rochkind
May 16, 2012, 2:21:45 PM5/16/12
to VCR Rubygem
Hi, I'm a newcomer to VCR. My code makes lots of HTTP requests to a
variety of third party services, so VCR is great.
I am using Test::Unit, which I'm not sure how usual that is as opposed
to rspec, but it works.
But I have some questions about best practices/golden path. I want to
do things how VCR wants,rather than fight it (unless i have a reason
to do things different), but i'm still figuring out what VCR wants.
#1
I've got _lots_ of tests that make third party HTTP calls, all of
which I want to wrap with VCR. In a test::unit suite, it seems that
every single one must be wrapped in a `use_cassette'.
test_something
VCR.use_cassette("something") do
# actual code
end
end
That's a whole lot of typing, for my many many tests. Additionally,
in the default :once configuration, every one of these needs to be a
_different_ cassette name too. That's a lot of cassettes.
Am I doing something wrong/weird, or is this how VCR's expecting
people to use it with Test::Unit? Do people using with rspec have
advantage of certain things that make this more tolerable, what ends
up happening there?
If this is how use is expected, I'm contemplating writing my own
wrapper method, something like:
test_with_cassette("tests_something") do
# actual test logic
end
That would automatically define the test_method for you based on the
name you pass in, wrap your actual test logic in a use_cassette, with
a a cassette name based on the name you pass in. Does this seem like
a sensible thing to do?
#2
My other issue is that, my code talks to a bunch of different third
party API's. Some of these require an API key or other auth. Some
collaborating developers may not in fact have access to an API key for
a given service. So VCR could be great here, allowing people to run
test code against recorded HTTP interactions even for services they
don't have access to (so long as the requests don't change). I see
how to use `filter_sensitive_data` to keep the private API keys out of
the cached requests/responses in the repo, great.
But there's one thing that's not entirely clear to me the best way to
do. I'll want to be able to erase tapes and re-record, just for one
particular foreign service. It seems that the way you erase tapes and
re-record with VCR is simply by erasing the cassette.yml on disk? But
especially with the cassette-per-test that, as above, it seems I'll be
using -- if I want to erase tapes just for a certain provider and not
all of them, that's a lot of cassettes to sort through to figure out
which to erase.
I looked around, but I don't _think_ there's any way to erase/re-
record just certain casettes based on a tag, right?
Is it possible to have cassetes be in sub-directories of your main
cassette_library_dir? Maybe I can arrange to have all the cassettes
for a particular service in a particular subdir, to make it easier to
erase em all, for a particular service. I guess my prior idea about
making a helper method to wrap my actual test methods in a
use_cassette might come in handy there too.
Thanks for any feedback or advice! I'm worried if I just hack
something out, I'll be fighting VCR when there's an easier way to do
it that VCR intends, really appreciate advice from folks who
understand VCR's "easy path" better than me.
variety of third party services, so VCR is great.
I am using Test::Unit, which I'm not sure how usual that is as opposed
to rspec, but it works.
But I have some questions about best practices/golden path. I want to
do things how VCR wants,rather than fight it (unless i have a reason
to do things different), but i'm still figuring out what VCR wants.
#1
I've got _lots_ of tests that make third party HTTP calls, all of
which I want to wrap with VCR. In a test::unit suite, it seems that
every single one must be wrapped in a `use_cassette'.
test_something
VCR.use_cassette("something") do
# actual code
end
end
That's a whole lot of typing, for my many many tests. Additionally,
in the default :once configuration, every one of these needs to be a
_different_ cassette name too. That's a lot of cassettes.
Am I doing something wrong/weird, or is this how VCR's expecting
people to use it with Test::Unit? Do people using with rspec have
advantage of certain things that make this more tolerable, what ends
up happening there?
If this is how use is expected, I'm contemplating writing my own
wrapper method, something like:
test_with_cassette("tests_something") do
# actual test logic
end
That would automatically define the test_method for you based on the
name you pass in, wrap your actual test logic in a use_cassette, with
a a cassette name based on the name you pass in. Does this seem like
a sensible thing to do?
#2
My other issue is that, my code talks to a bunch of different third
party API's. Some of these require an API key or other auth. Some
collaborating developers may not in fact have access to an API key for
a given service. So VCR could be great here, allowing people to run
test code against recorded HTTP interactions even for services they
don't have access to (so long as the requests don't change). I see
how to use `filter_sensitive_data` to keep the private API keys out of
the cached requests/responses in the repo, great.
But there's one thing that's not entirely clear to me the best way to
do. I'll want to be able to erase tapes and re-record, just for one
particular foreign service. It seems that the way you erase tapes and
re-record with VCR is simply by erasing the cassette.yml on disk? But
especially with the cassette-per-test that, as above, it seems I'll be
using -- if I want to erase tapes just for a certain provider and not
all of them, that's a lot of cassettes to sort through to figure out
which to erase.
I looked around, but I don't _think_ there's any way to erase/re-
record just certain casettes based on a tag, right?
Is it possible to have cassetes be in sub-directories of your main
cassette_library_dir? Maybe I can arrange to have all the cassettes
for a particular service in a particular subdir, to make it easier to
erase em all, for a particular service. I guess my prior idea about
making a helper method to wrap my actual test methods in a
use_cassette might come in handy there too.
Thanks for any feedback or advice! I'm worried if I just hack
something out, I'll be fighting VCR when there's an easier way to do
it that VCR intends, really appreciate advice from folks who
understand VCR's "easy path" better than me.
Myron Marston
May 16, 2012, 2:27:03 PM5/16/12
to VCR Rubygem
I'll try to get around to it tonight or tomorrow. In the meantime,
maybe someone else can offer some advice!
Myron Marston
May 17, 2012, 1:46:55 AM5/17/12
to VCR Rubygem
> I've got _lots_ of tests that make third party HTTP calls, all of
> which I want to wrap with VCR. In a test::unit suite, it seems that
> every single one must be wrapped in a `use_cassette'.
> which I want to wrap with VCR. In a test::unit suite, it seems that
> every single one must be wrapped in a `use_cassette'.
> That's a whole lot of typing, for my many many tests.
VCR makes it easy to make virtually all your tests integration tests,
where HTTP requests are made all over the place, but I'd encourage you
not to do this. Instead, try to limit the number of tests that need
to use VCR by being limiting it to two kinds of tests:
* Wrap all 3rd party API access in objects you own. Make these
objects have a simple, well-defined interface that speaks in the
domain of your system. When you test these objects, use VCR.
* Use VCR for full-stack acceptance tests.
For all the other tests (e.g. tests of objects that collaborate w/
your 3rd party API wrappers) use a test double in place of your 3rd
party API wrappers. If you design them properly, with a small, simple
interface, this should be relatively easy to do.
For more on this topic, I encourage you to check out these resources:
* https://github.com/craigw/craigw.github.com/blob/79e78a800907adb5d7c0a44af6484f1ca5f03510/_drafts/2012-04-10-my-thoughts-on-vcr.md
* The sucks/rock series on destroy all software, particularly
https://www.destroyallsoftware.com/screencasts/catalog/sucks-rocks-3-the-search-engine
> Additionally,
> in the default :once configuration, every one of these needs to be a
> _different_ cassette name too. That's a lot of cassettes.
as long as they make the same sequence of requests. In general, it's
a good idea to map one cassette to one sequence of requests.
> Am I doing something wrong/weird, or is this how VCR's expecting
> people to use it with Test::Unit? Do people using with rspec have
> advantage of certain things that make this more tolerable, what ends
> up happening there?
or should not use VCR. The main API is `VCR.use_cassette` (as your
example uses) because this is a general API that is coupled to
_nothing_ (and can work outside of a test framework, in fact) and can
work _anywhere_. However, as you point out, it can be verbose to
manually use this everywhere. High-level abstractions are certainly
possible, and in fact encouraged. VCR itself ships with a couple of
the higher-level abstractions in the form of the RSpec and Cucumber
integration:
* https://www.relishapp.com/myronmarston/vcr/v/2-1-1/docs/test-frameworks/usage-with-rspec-metadata
* https://www.relishapp.com/myronmarston/vcr/v/2-1-1/docs/test-frameworks/usage-with-rspec-macro
* https://www.relishapp.com/myronmarston/vcr/v/2-1-1/docs/test-frameworks/usage-with-cucumber
This is possible because RSpec and cucumber (and especially RSpec)
provide a rich API full of extension points for something like VCR to
hook into in a generic way. Test::Unit does not provide any similar
set of hooks, so there's no way for VCR to hook into it in a generic
way, and you're essentially on your own there.
The `test_with_cassette` method you pasted looks pretty reasonable,
but you may also want to consider writing helper methods are higher-
level versions of `VCR.use_cassette` specific to your domain. For
example, if your app made lots of API requests to github and did stuff
with repositories a lot, you might want a helper method like:
def use_github_cassette(user, repo)
VCR.use_cassette("github/#{user}/#{repo}") do
yield # or maybe even do something here to make the request
directly
end
end
> It seems that the way you erase tapes and
> re-record with VCR is simply by erasing the cassette.yml on disk?
record mode to :all or use the :re_record_interval option.
> I looked around, but I don't _think_ there's any way to erase/re-
> record just certain casettes based on a tag, right?
> Is it possible to have cassetes be in sub-directories of your
> main cassette_library_dir
will be "foo/bar/bazz.yml" relative to the cassette_library_dir. I
use this all the time to arrange my cassettes.
HTH,
Myron
Jonathan Rochkind
May 17, 2012, 10:54:14 AM5/17/12
to vcr-...@googlegroups.com
Cool, thanks, this was very helpful.
The nature of my app.... or maybe it's my programming style or way of
thinking too.... makes it difficult to use lots of test doubles, I
think. Or maybe even if so....
Yes, I'm wrapping all 3rd party API access in objects I own. But the
thing is, there are a LOT of these objects I own -- in fact, it's kind
of the primary function of this library to provide such objects,
abstracting _many_ 3rd party API's. (You can see why I wound up
investigating VCR). So there's a bunch of tests of actual functionality
of my "own" wrapper objects -- even if I use test doubles everywhere
else (which I find to be a pain, but that might be my fault).
So anyhow, I'm still leaning toward making my own wrapper method. Model
it on what Rails does with Test::Unit, to dynamically create test
methods. (This is the one thing you CAN do relatively easily with
Test::Unit or minitest).
I'm trying to understand what VCR is doing in your relish rspec/cucumber
examples -- the rspec example has no use_cassette but still records with
VCR --- VCR just automatically wraps everything in a use_cassette? Ah,
it looks like a cassette by default named after the example group/example?
What I'm thinking is:
test_with_cassette("does something", :group, optional_extra_args) do
assert test_logic
end
Would:
* Define a test method based on "does something".
* Wrap it in a use_cassette with cassette name based on "does
something", but with a subdir based on :group.
* hey, add a :tag based on :group to the use_cassette too, so you can
use it for filter_sensitive_data.
* pass on any optional_extra_args to use_cassette too.
Now I can write things concisely. I get all cassettes in a particular
:group in a subdir, so I can easily erase them all at once-- but no
cassettes from different groups -- without changing any checked in
source _but_ for deleting the cassettes (no need to change configuration
in checked in source).
This _seems_ like it will work well for me.
But please do say if you think I'm heading down a mistaken path. I'm
thinking atm that this to some extent simulates what you can do more
easily with rspec, the same sort of use patterns, but without the
ability not available outside of rspec to automatically pick cassette
name from example group/example, or set cassette name for an entire
example group (at some point I may look into if either of those can be
done for minitest/spec).
I _think_ 'back to basics' test::unit/minitest is catching on a bit
more, if I work something out like this that seems useful, perhaps I'll
add it to the VCR wiki as an example of one way to approach test::unit
with VCR?
The nature of my app.... or maybe it's my programming style or way of
thinking too.... makes it difficult to use lots of test doubles, I
think. Or maybe even if so....
Yes, I'm wrapping all 3rd party API access in objects I own. But the
thing is, there are a LOT of these objects I own -- in fact, it's kind
of the primary function of this library to provide such objects,
abstracting _many_ 3rd party API's. (You can see why I wound up
investigating VCR). So there's a bunch of tests of actual functionality
of my "own" wrapper objects -- even if I use test doubles everywhere
else (which I find to be a pain, but that might be my fault).
So anyhow, I'm still leaning toward making my own wrapper method. Model
it on what Rails does with Test::Unit, to dynamically create test
methods. (This is the one thing you CAN do relatively easily with
Test::Unit or minitest).
I'm trying to understand what VCR is doing in your relish rspec/cucumber
examples -- the rspec example has no use_cassette but still records with
VCR --- VCR just automatically wraps everything in a use_cassette? Ah,
it looks like a cassette by default named after the example group/example?
What I'm thinking is:
test_with_cassette("does something", :group, optional_extra_args) do
assert test_logic
end
Would:
* Define a test method based on "does something".
* Wrap it in a use_cassette with cassette name based on "does
something", but with a subdir based on :group.
* hey, add a :tag based on :group to the use_cassette too, so you can
use it for filter_sensitive_data.
* pass on any optional_extra_args to use_cassette too.
Now I can write things concisely. I get all cassettes in a particular
:group in a subdir, so I can easily erase them all at once-- but no
cassettes from different groups -- without changing any checked in
source _but_ for deleting the cassettes (no need to change configuration
in checked in source).
This _seems_ like it will work well for me.
But please do say if you think I'm heading down a mistaken path. I'm
thinking atm that this to some extent simulates what you can do more
easily with rspec, the same sort of use patterns, but without the
ability not available outside of rspec to automatically pick cassette
name from example group/example, or set cassette name for an entire
example group (at some point I may look into if either of those can be
done for minitest/spec).
I _think_ 'back to basics' test::unit/minitest is catching on a bit
more, if I work something out like this that seems useful, perhaps I'll
add it to the VCR wiki as an example of one way to approach test::unit
with VCR?
Myron Marston
May 17, 2012, 11:15:25 AM5/17/12
to VCR Rubygem
> The nature of my app.... or maybe it's my programming style or way of
> thinking too.... makes it difficult to use lots of test doubles, I
> think. Or maybe even if so....
One of the values of test doubles is that when they are difficult,
> thinking too.... makes it difficult to use lots of test doubles, I
> think. Or maybe even if so....
they give you valuable design feedback and can drive you to a design
with simpler interfaces. It's generally easy to use a test double for
a collaborator that has a small, simple interface. It's hard to use a
test double for a collaborator with a large or complex interface.
That said, in practice it's obviously not so black and white, and it's
ultimately about trade offs :).
> I'm trying to understand what VCR is doing in your relish rspec/cucumber
> examples -- the rspec example has no use_cassette but still records with
> VCR --- VCR just automatically wraps everything in a use_cassette? Ah,
> it looks like a cassette by default named after the example group/example?
scenario based on tags. There's a little `VCR.cucumber_tags` API for
specifying them. The tag name is used as the cassette name by
default.
There's two ways the rspec integration works; one is a macro
(`use_vcr_cassette`) that adds before/after hooks to the example
groups to have it use the same cassette for each example in the
group. The newer support is for RSpec 2.x only and uses RSpec
metadata. So you can do this...
it "does something", :vcr
...and it will use a cassette named after the example and example
group. You can also provide a hash of options:
it "does something", vcr: { match_requests_on: [:uri, :body],
cassette_name: 'foo' }
...and it will use the passed cassette options. The `cassette_name`
option here isn't a normal option; it's an added one for this context
that lets the user override what the cassette name is so that they can
use the same cassette for multiple examples.
> What I'm thinking is:
>
> test_with_cassette("does something", :group, optional_extra_args) do
> assert test_logic
> end
the RSpec metadata approach I mentioned above, you could tweak this to
work similarly:
test("does something", :vcr) do
end
test("does something else", :vcr => { ... }) do
end
One benefit here is that if/when you have additional test metadata or
"wrappings" (e.g. VCR.use_cassette { }) you want to do, you can just
support more hash keys. But what you have is totally fine and it
sounds like it actually may fit your use case a bit better.
> I _think_ 'back to basics' test::unit/minitest is catching on a bit
> more, if I work something out like this that seems useful, perhaps I'll
> add it to the VCR wiki as an example of one way to approach test::unit
> with VCR?
added:
https://github.com/myronmarston/vcr/wiki/Usage-with-MiniTest
Myron
> > * The sucks/rock series on destroy all software, particularly
> >https://www.destroyallsoftware.com/screencasts/catalog/sucks-rocks-3-...
>
> >> Additionally,
> >> in the default :once configuration, every one of these needs to be a
> >> _different_ cassette name too. That's a lot of cassettes.
>
> > That's actually not true. Multiple tests can re-use the same cassette
> > as long as they make the same sequence of requests. In general, it's
> > a good idea to map one cassette to one sequence of requests.
>
> >> Am I doing something wrong/weird, or is this how VCR's expecting
> >> people to use it with Test::Unit? Do people using with rspec have
> >> advantage of certain things that make this more tolerable, what ends
> >> up happening there?
>
> > In general, VCR makes relatively few expectations about how you should
> > or should not use VCR. The main API is `VCR.use_cassette` (as your
> > example uses) because this is a general API that is coupled to
> > _nothing_ (and can work outside of a test framework, in fact) and can
> > work _anywhere_. However, as you point out, it can be verbose to
> > manually use this everywhere. High-level abstractions are certainly
> > possible, and in fact encouraged. VCR itself ships with a couple of
> > the higher-level abstractions in the form of the RSpec and Cucumber
> > integration:
>
> > *https://www.relishapp.com/myronmarston/vcr/v/2-1-1/docs/test-framewor...
> >> Additionally,
> >> in the default :once configuration, every one of these needs to be a
> >> _different_ cassette name too. That's a lot of cassettes.
>
> > That's actually not true. Multiple tests can re-use the same cassette
> > as long as they make the same sequence of requests. In general, it's
> > a good idea to map one cassette to one sequence of requests.
>
> >> Am I doing something wrong/weird, or is this how VCR's expecting
> >> people to use it with Test::Unit? Do people using with rspec have
> >> advantage of certain things that make this more tolerable, what ends
> >> up happening there?
>
> > In general, VCR makes relatively few expectations about how you should
> > or should not use VCR. The main API is `VCR.use_cassette` (as your
> > example uses) because this is a general API that is coupled to
> > _nothing_ (and can work outside of a test framework, in fact) and can
> > work _anywhere_. However, as you point out, it can be verbose to
> > manually use this everywhere. High-level abstractions are certainly
> > possible, and in fact encouraged. VCR itself ships with a couple of
> > the higher-level abstractions in the form of the RSpec and Cucumber
> > integration:
>
> > *https://www.relishapp.com/myronmarston/vcr/v/2-1-1/docs/test-framewor...
> > *https://www.relishapp.com/myronmarston/vcr/v/2-1-1/docs/test-framewor...
Jonathan Rochkind
May 17, 2012, 11:36:15 AM5/17/12
to vcr-...@googlegroups.com
Woah, I didn't notice that already existing VCR minitest/spec wiki page!
That's super helpful, thanks!
I may actually change my tests over to minitest/spec at some point
soonish, in part motivated by that. (I was thinking I'd have to figure
out how to do exactly what your wiki page demonstrates myself).
But I'll probably add my test::unit example anyway.
Might make sense to at least mention in the README "Check out the wiki
for some other ways of doing minitest and test::unit integration."
VCR comes with pretty powerful/flexible built-in support for
Rspec/cucumber -- but for test::unit/minispec, I think users are
_likely_ to need something along the lines of your minitest wiki page or
what i'm going to do for test::unit.
(If we knew that many users would need pretty much _just_ what's in the
wiki, I'd suggest it should be added to VCR source by default. But maybe
the test-unit/minitest VCR situation is still like, eh, not sure, some
people may use this, some people may have to modify it somewhat for what
they need, many may not want it at all. In which case the wiki makes
sense).
That's super helpful, thanks!
I may actually change my tests over to minitest/spec at some point
soonish, in part motivated by that. (I was thinking I'd have to figure
out how to do exactly what your wiki page demonstrates myself).
But I'll probably add my test::unit example anyway.
Might make sense to at least mention in the README "Check out the wiki
for some other ways of doing minitest and test::unit integration."
VCR comes with pretty powerful/flexible built-in support for
Rspec/cucumber -- but for test::unit/minispec, I think users are
_likely_ to need something along the lines of your minitest wiki page or
what i'm going to do for test::unit.
(If we knew that many users would need pretty much _just_ what's in the
wiki, I'd suggest it should be added to VCR source by default. But maybe
the test-unit/minitest VCR situation is still like, eh, not sure, some
people may use this, some people may have to modify it somewhat for what
they need, many may not want it at all. In which case the wiki makes
sense).
Jonathan Rochkind
May 17, 2012, 11:44:08 AM5/17/12
to vcr-...@googlegroups.com
In fact, you know why I didn't notice that wiki page on minitest? I now
an reminded seeing it again.
Cause I went to the wiki and found this message:
"Note: this wiki is not actively maintained. The best source of VCR
documentation is the relish site."
I then abandoned the wiki, not noticing/paying attention to:
"Feel free to edit the wiki to add any tips/tricks you have discovered
for using VCR."
(Or figuring nobody probably had).
Can I change the wiki home page? If there are any wiki pages that
duplicate what's in relish, let's just delete them? And then we can
leave wiki pages that do not duplicate what's in relish for whatever
reason, and make the home page not say 'abandoned', but just say 'best
documetnation is relish, this wiki contains user-contributed tips and
tricks and extensions."
an reminded seeing it again.
Cause I went to the wiki and found this message:
"Note: this wiki is not actively maintained. The best source of VCR
documentation is the relish site."
I then abandoned the wiki, not noticing/paying attention to:
"Feel free to edit the wiki to add any tips/tricks you have discovered
for using VCR."
(Or figuring nobody probably had).
Can I change the wiki home page? If there are any wiki pages that
duplicate what's in relish, let's just delete them? And then we can
leave wiki pages that do not duplicate what's in relish for whatever
reason, and make the home page not say 'abandoned', but just say 'best
documetnation is relish, this wiki contains user-contributed tips and
tricks and extensions."
Myron Marston
May 17, 2012, 12:41:54 PM5/17/12
to VCR Rubygem
> > (If we knew that many users would need pretty much _just_ what's in the
> > wiki, I'd suggest it should be added to VCR source by default. But maybe
> > the test-unit/minitest VCR situation is still like, eh, not sure, some
> > people may use this, some people may have to modify it somewhat for what
> > they need, many may not want it at all. In which case the wiki makes
> > sense).
I'd be happy to accept a pull request adding a "usage with minitest"
> > wiki, I'd suggest it should be added to VCR source by default. But maybe
> > the test-unit/minitest VCR situation is still like, eh, not sure, some
> > people may use this, some people may have to modify it somewhat for what
> > they need, many may not want it at all. In which case the wiki makes
> > sense).
cuke (the shoulda and test/unit cukes should get you started if you
decide to take a stab at this).
> "Note: this wiki is not actively maintained. The best source of VCR
> documentation is the relish site."
the README. As it grew, the README became unwieldy, and I decided to
move it to separate pages on the wiki. Later, I found out about
relish, and decided to use my cukes as documentation. The wiki got
left as-is and became quickly out of date. So then I removed the old
out-of-date stuff and put the "not actively maintained" note on the
wiki since I'm not using it anymore for official VCR documentation.
> Can I change the wiki home page? If there are any wiki pages that
> duplicate what's in relish, let's just delete them? And then we can
> leave wiki pages that do not duplicate what's in relish for whatever
> reason, and make the home page not say 'abandoned', but just say 'best
> documetnation is relish, this wiki contains user-contributed tips and
> tricks and extensions."
editable. I can always revert changes if I don't agree with them but
I doubt that would happen.
Myron
>
> read more »
Jonathan Rochkind
May 17, 2012, 6:16:58 PM5/17/12
to vcr-...@googlegroups.com
Okay, I'm gonna test more before I add to the wiki (and, um, maybe write
some tests. I did not do this TDD; writing tests for a testing framework
is blowing my mind, although of course it's what you've done)
Probably wont' get to that until next week.
But the draft works, it's nice, it excites me, I want to share.
https://gist.github.com/2721883
Thanks for VCR man, it's awesome.
some tests. I did not do this TDD; writing tests for a testing framework
is blowing my mind, although of course it's what you've done)
Probably wont' get to that until next week.
But the draft works, it's nice, it excites me, I want to share.
https://gist.github.com/2721883
Thanks for VCR man, it's awesome.
Reply all
Reply to author
Forward
0 new messages