Stub network access in shell code

[RocketR]

Hi,
I use AffiliateWindow service to import a feed of products for my Rails app. The feed is a compressed xml file, which can reach hundreds of megabytes in size, so I open a pipe in my app to `curl | gunzip` to parse it sequentially. Now, it works fine and I have an integration test that downloads the feed(a filtered down version), parses and stores the products.

The problem is that every other day it fails because some products disappear from the feed. What can I do about this? Something like a VCR for shell would be ideal. Or should I somehow create a fake service that starts along with the test at some url and returns the hard-coded feed?

Thanks

[Dan Rabinowitz]

Just wrap the call to curl in a method call. Then you can stub the method call.

Or maybe use the "curb" gem, although I don't know anything about it.

-dan

[Steve Tooke]

On 5 March 2014 18:37, RocketR <broil...@gmail.com> wrote:

I use AffiliateWindow service to import a feed of products for my Rails app. The feed is a compressed xml file, which can reach hundreds of megabytes in size, so I open a pipe in my app to `curl | gunzip` to parse it sequentially. Now, it works fine and I have an integration test that downloads the feed(a filtered down version), parses and stores the products.

The problem is that every other day it fails because some products disappear from the feed. What can I do about this? Something like a VCR for shell would be ideal. Or should I somehow create a fake service that starts along with the test at some url and returns the hard-coded feed?

If the integration point is a pipe to an external command, could you make the command a variable and replace `curl | gunzip` with `cat` or even `echo`for testing?

[RocketR]

I'd like to have this test as close to end-to-end as possible. I can't stub a method that creates a pipe or replace `curl` with `cat` because then I won't be testing if the URL is built correctly and if the shell command works at all.

[Dave Newton]

On Wed, Mar 5, 2014 at 5:18 PM, RocketR <broil...@gmail.com> wrote:

I'd like to have this test as close to end-to-end as possible. I can't stub a method that creates a pipe or replace `curl` with `cat` because then I won't be testing if the URL is built correctly and if the shell command works at all.

Those sound like totally separate tests to me, along with a test that exercises "disappearing product" behavior.

Dave

[Roman Kushnir]

I have separate detailed unit tests for each part, but also need a test to prove that everything works together.

Best Regards,
Roman

--
You received this message because you are subscribed to a topic in the Google Groups "Objects on Rails" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/objects-on-rails/lmQv4LxXxnU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to objects-on-rai...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Dave Newton]

Then what's the issue with the original answer? If you already test that the URL is built correctly, that "disappearing product" behavior works, etc. then the issue is that you want an integration test that uses your own URL that returns fake data. Things like FakeWeb or a simple fake service endpoint you can spin up is enough.

I don't yet understand what the issue is; I'm sorry.

Dave

--
You received this message because you are subscribed to the Google Groups "Objects on Rails" group.
To unsubscribe from this group and stop receiving emails from it, send an email to objects-on-rai...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

e: davel...@gmail.com

m: 908-380-8699

s: davelnewton_skype

t: @dave_newton

b: Bucky Bits

g: davelnewton

so: Dave Newton

[RocketR]

My understanding is that purpose of integration tests expects no (or at least minimal) use of stubbing/mocking. If I just stub `download` method to return some local file how do I know if it works for real?

[Dave Newton]

... Your "integration test" sounds more like "running the program".

You'll know it doesn't work if running the program fails. Your tests should mimic the possible responses from the external service: network issues, response data, etc.

Once an integration test crosses the boundaries past what you can actually control it's not a test, it's the real thing.

[mike nicholaides]

If you want to make your integration test more realistic, you can spin up Webbrick to serve an example version of the file and make the URL that your program hits configurable between environments.

Mike Nicholaides

Software Engineer and Co-Founder at PromptWorks

promptworks.com

(267) 622-4729

[Nicolás Sanguinetti]

In line with that suggestion, I've used https://github.com/djanowski/mock-server a bunch in the past.

[Cynthia Kiser]

Quoting Roman Kushnir <broil...@gmail.com>:
> I have separate detailed unit tests for each part, but also need a test to
> prove that everything works together.

How about changing the test such that it tests the structure of the
XML that comes back, rather than testing the data that is in the
feed. Your integration test will then tell you are able to retrieve
and parse the feed and that the feed contains something
reasonable. But the details of what gets returned is out of scope.

--
Cynthia N. Kiser
c...@ugcs.caltech.edu

[Roman Kushnir]

I think I'll give the local webrick server a try, it seems the least intrusive as requires only changing the hostname in config. Thank you everyone for the answers.

[mike nicholaides]

At PromptWorks we have a test helper class that we've been meaning to gemify for a while. It spins up a server (or any process), and waits for it to start listening on a port before continuing with the test. We use it often for external services that the integration tests rely on.

If you'd like I can throw up a gist with instructions. Or, if someone knows of something like this that's already in a gem maybe I'll start using that instead.

Mike Nicholaides

Software Engineer and Co-Founder at PromptWorks

promptworks.com

(267) 622-4729

--

You received this message because you are subscribed to the Google Groups "Objects on Rails" group.

To unsubscribe from this group and stop receiving emails from it, send an email to objects-on-rai...@googlegroups.com.

[Chris McGrath]

I’ve used RealWeb for things like this before

https://github.com/engineyard/realweb

HTH,

Chris

For more options, visit https://groups.google.com/d/optout.

[RocketR]

Thanks. I tried realweb and it does the job! Only caveat is I couldn't find a way to correctly handle ActiveRecord's connect/disconnect when using a forking server, it breaks the transactional fixtures, so I had to use a threaded server. Maybe a better idea would be to spawn a new process instead of forking.