fog and VCR

[Kevin Menard]

Hi,

VCR has an excon adapter, which I was hoping would let me seemlessly
mock out S3 connections much like we do for other HTTP calls.
Unfortunately, it doesn't seem to be working and I think it's related
to the mocked response from excon not triggering fog's SAX parser.
I'd be happy to work up a patch if that's the case, but before I do,
has anyone gotten VCR working with fog?

Thanks,
Kevin

[geemus (Wesley Beary)]

I wonder why you want to do it this way, rather than using fog's built
in mocking? You should be able to just call Fog.mock! and have mocked
S3 interactions pretty seamlessly. If that isn't enough to support
your use case, let me know though and we'll try to figure out how to
piece it all together.

[Kevin Menard]

I'd like to work with live data over real transactions.  I could just use mocha to mock otherwise.

VCR has the added benefit of being very unobtrusive from the caller's perspective.  I just insert the cassette and can seamlessly test anything that makes outgoing HTTP calls.  We used to be able to do this just fine when we use aws-s3 but had to give it up when switching to fog.

A tertiary benefit is it's much easier to debug when the request and response are logged to a YAML file.  There are other tools for doing that, but it's nifty nonetheless.

To clarify, the use case I'm looking to handle immediately is interactions with S3, but I could see value in doing it for other types of calls, too.

--
Kevin

---

geemus (Wesley Beary) June 3, 2011 5:55 PM

I wonder why you want to do it this way, rather than using fog's built
in mocking? You should be able to just call Fog.mock! and have mocked
S3 interactions pretty seamlessly. If that isn't enough to support
your use case, let me know though and we'll try to figure out how to
piece it all together.

Kevin Menard June 2, 2011 4:56 PM

[geemus (Wesley Beary)]

I guess I still don't understand how Fog.mock! fails to provide for that use case.  It should take your real data and do the right thing in most all cases (and is what other people with similar use cases have used in so far as I'm aware).  Have you tried it and run into edge cases or are there particular things that it misses?

Thanks!
wes

[Kevin Menard]

Hi Wesley,

I just wanted to follow back-up.  First, sorry the long delay, but I was AFK for about a week and a half.

So I went back through and did everything with fog's mocking framework and it wasn't as bad as I thought.  At the core of it, I'm not a fan of mocked replies given how badly I've been burned in the past.  E.g., it's quite possible there's a bug in one of fog's mocked replies and using something like VCR would guard against that.  I mentally got over that by realizing a bug in the mocking code almost certainly means there's a corresponding bug in fog itself.

A few other reasons why I prefer using something like VCR:

- Easier to replace fog if need be.  In fact, we already had a test suite built up from aws-s3 that I ended up trashing, when I should have theoretically been able to reuse the VCR cassettes.

- It's a lot faster to build up a test bucket containing the test data than it is to create them in the test.

- Fog mocking doesn't mimic real world performance behaviors because the SAX parsers are completely skipped from what I can tell.  This was a real problem for us.  In writing a utility to calculate the size of the bucket we found it almost three orders of magnitude faster to use straight requests rather than the models.

Anyway, things are working out well now.  With one exception that I'll raise a separate issue for.

Independently of all that, and looping back to the original issue, it's still a bit confusing that VCR supports Excon and fog uses Excon.  At the very least it might help others such as myself to have a note in the documentation somewhere that this isn't a supported scenario.

Thanks again.  Fog is truly one the best put together Ruby projects I've ever come across.  I look forward to using it out on other projects.

--
Kevin

---

geemus (Wesley Beary) June 3, 2011 6:11 PM

I guess I still don't understand how Fog.mock! fails to provide for that use case.  It should take your real data and do the right thing in most all cases (and is what other people with similar use cases have used in so far as I'm aware).  Have you tried it and run into edge cases or are there particular things that it misses?

Thanks!
wes

---

Kevin Menard June 3, 2011 6:07 PM

[geemus (Wesley Beary)]

Glad to hear it all worked out, and thank you for you specific responses as to your concerns.

I do run the tests in fog against the mocked as well as real modes, so as you said, a bug in the mocking code is probably a bug in fog itself (fwiw).

There definitely is some overheard to switching between libraries that vcr might help accomodate (I've made this switch myself in other places, also without vcr, and I can see how it would benefit).

I wasn't aware that vcr took latency into account, would using vcr have helped with the performance issues, vs the mocks?

The support for VCR was added to excon as a separate concern from fog (and fairly recently).  It is there to provide, mostly, for people who are using excon for other things.  That said, I'm not trying to say that it shouldn't work.  It is just not something I have gone to any length to provide for (since it seems like there is a ready solution with the mocks).  So, I am glad you got the mocks working and I think that in most cases they will be easier to use, but it sounds like there are some reasons why you might want to opt for more difficult (but more accurate) usage of vcr directly.  I'll add it to the list of things to think about going forward.  So far you have been the first to really bring it up though, everyone else seems to have been fine with the mocks (or perhaps they are not testing well or doing something else and I'm not aware of it).  Would you still want vcr support to be added in fog in the future or do you find that in the end the mocks do a good enough job supporting your use case?

Also, out of curiousity, is there anything in particular that lead you toward trying to approach this via VCR instead of the fog mocks?  I'd like to shore up the documentation to make sure the mocked path is more clearly the recommended one (at least for the time being).

Thanks again for all the feedback, its always valuable to me when I can hear more specifically about use cases and concerns.

wes

[Kevin Menard]

Hi Wesley,

Please find responses inline:

---

geemus (Wesley Beary) June 15, 2011 12:05 PM

I wasn't aware that vcr took latency into account, would using vcr have helped with the performance issues, vs the mocks?

Having not been able to successfully parse a mocked excon response via VCR, I can't really say for certain.  If the SAX parsing step was skipped, then it almost certainly wouldn't have highlighted any performance issues.

The support for VCR was added to excon as a separate concern from fog (and fairly recently).  It is there to provide, mostly, for people who are using excon for other things.  That said, I'm not trying to say that it shouldn't work.  It is just not something I have gone to any length to provide for (since it seems like there is a ready solution with the mocks).  So, I am glad you got the mocks working and I think that in most cases they will be easier to use, but it sounds like there are some reasons why you might want to opt for more difficult (but more accurate) usage of vcr directly.  I'll add it to the list of things to think about going forward.  So far you have been the first to really bring it up though, everyone else seems to have been fine with the mocks (or perhaps they are not testing well or doing something else and I'm not aware of it).  Would you still want vcr support to be added in fog in the future or do you find that in the end the mocks do a good enough job supporting your use case?

I'm happy with the mocks.  It's infinitely better than doing it with mocha, which we're trying to phase out completely since it's failed us in every non-trivial refactoring we've performed.  Having looked through the fog mock code, I'm pretty confident we wouldn't be susceptible to the same problems.  My largest argument for VCR support at this point is just that it seems like it should work.  I realize all that is nascent, but I thought for certain I was doing something wrong and it just turns out that the way the mock response is returned in Excon just doesn't work with fog.  Adding a caveat message to the docs might be sufficient.

Also, out of curiousity, is there anything in particular that lead you toward trying to approach this via VCR instead of the fog mocks?  I'd like to shore up the documentation to make sure the mocked path is more clearly the recommended one (at least for the time being).

The AWS APIs are HTTP.  When we test HTTP, we use VCR.  It's been the best approach for us, yielding high quality tests that actually highlight failures when something breaks (e.g., bad credentials).  We also use the ERB support in VCR a fair bit in general just so we can cope with requests & responses using nonces.  To me, the fact fog was using excon was simply an implementation detail.  I figured it was using HTTP, so I turned to our HTTP testing library.

I should add that a fair bit of this can be attributed to my ignorance of fog's mocking stuff.  I didn't quite understand what it was and it wasn't until I ran through the tutorial and then studied the library's code that I was comfortable with it.  I don't recall the tutorial having been available when I first started with fog several minor versions back, so this is a great addition.

Thanks again for all the feedback, its always valuable to me when I can hear more specifically about use cases and concerns.

No problem.  Thanks for not outright dismissing me :-)

Regards,
Kevin

[geemus (Wesley Beary)]

Skipping the SAX parsers is mostly a matter of convenience.  Especially prior to excon supporting VCR, mocking at this slightly higher level seemed much more manageable and maintainable than building/parsing xml in some way shape or form.  Granted this does mean that some things won't be tested in mocked mode, but it seems like a reasonable trade off in most cases.

I'm very glad that the mocks have worked out for you.  Sorry if I seemed dismissive or anything when we started the conversation, I just hadn't heard anybody ask about this. Plus I wasn't sure why you wanted it, though I know have a good idea.  It would be lovely if it could work as an alternative to the current mocks, but as I said, this was a tradeoff in terms of ease of development and use that seems reasonable in most cases.  It sounds like the docs have helped flesh this out since when you looked before, which I'm also glad to hear (they have been a major focus for me in the recent past and I've tried very hard to make up for former shortcomings).

Thanks again for the extra info and perspective.  I've added it to my notes to consider in the future, but it probably won't be too high of a priority (since you have another solution now and others haven't brought it up).

Thanks,

wes